1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
|
# Needle Layout Format
This document describes the binary layout of the Needle structure as used in SeaweedFS storage, for all supported versions (v1, v2, v3).
A Needle represents a file or data blob stored in a volume file. The layout determines how the Needle is serialized to disk for efficient storage and retrieval.
---
## Common Field Sizes
| Field | Size (bytes) |
|---------------|--------------|
| Cookie | 4 |
| NeedleId | 8 |
| Size | 4 |
| DataSize | 4 |
| Flags | 1 |
| NameSize | 1 |
| MimeSize | 1 |
| LastModified | 5 |
| Ttl | 2 |
| PairsSize | 2 |
| Checksum | 4 |
| Timestamp | 8 |
---
## Needle Layouts by Version
### Version 1
| Offset | Field | Size (bytes) | Description |
|----------|----------|--------------|-----------------------------------------------|
| 0 | Cookie | 4 | Random number to mitigate brute force lookups |
| 4 | Id | 8 | Needle ID |
| 12 | Size | 4 | Length of Data |
| 16 | Data | N | File data (N = Size) |
| 16+N | Checksum | 4 | CRC32 of Data |
| 20+N | Padding | 0-7 | To align to 8 bytes |
### Version 2
| Offset | Field | Size (bytes) | Description |
|----------|-------------|--------------|-----------------------------------------------|
| 0 | Cookie | 4 | Random number |
| 4 | Id | 8 | Needle ID |
| 12 | Size | 4 | Total size of the following fields |
| 16 | DataSize | 4 | Length of Data (N) |
| 20 | Data | N | File data |
| 20+N | Flags | 1 | Bit flags |
| 21+N | NameSize | 1 (opt) | Optional, if present |
| 22+N | Name | M (opt) | Optional, if present (M = NameSize) |
| ... | MimeSize | 1 (opt) | Optional, if present |
| ... | Mime | K (opt) | Optional, if present (K = MimeSize) |
| ... | LastModified| 5 (opt) | Optional, if present |
| ... | Ttl | 2 (opt) | Optional, if present |
| ... | PairsSize | 2 (opt) | Optional, if present |
| ... | Pairs | P (opt) | Optional, if present (P = PairsSize) |
| ... | Checksum | 4 | CRC32 |
| ... | Padding | 0-7 | To align to 8 bytes |
### Version 3
| Offset | Field | Size (bytes) | Description |
|----------|-------------|--------------|-----------------------------------------------|
| 0 | Cookie | 4 | Random number |
| 4 | Id | 8 | Needle ID |
| 12 | Size | 4 | Total size of the following fields |
| 16 | DataSize | 4 | Length of Data (N) |
| 20 | Data | N | File data |
| 20+N | Flags | 1 | Bit flags |
| 21+N | NameSize | 1 (opt) | Optional, if present |
| 22+N | Name | M (opt) | Optional, if present (M = NameSize) |
| ... | MimeSize | 1 (opt) | Optional, if present |
| ... | Mime | K (opt) | Optional, if present (K = MimeSize) |
| ... | LastModified| 5 (opt) | Optional, if present |
| ... | Ttl | 2 (opt) | Optional, if present |
| ... | PairsSize | 2 (opt) | Optional, if present |
| ... | Pairs | P (opt) | Optional, if present (P = PairsSize) |
| ... | Checksum | 4 | CRC32 |
| ... | Timestamp | 8 | Append time in nanoseconds |
| ... | Padding | 0-7 | To align to 8 bytes |
- Offsets marked with `...` depend on the presence and size of previous optional fields.
- Fields marked `(opt)` are optional and only present if the corresponding size or flag is non-zero.
- N = DataSize, M = NameSize, K = MimeSize, P = PairsSize.
---
## Field Explanations
- **Cookie**: 4 bytes, random value for security.
- **Id**: 8 bytes, unique identifier for the Needle.
- **Size**: 4 bytes, total size of the Needle data section (not including header, checksum, timestamp, or padding).
- **DataSize**: 4 bytes, length of the Data field.
- **Data**: File data (variable length).
- **Flags**: 1 byte, bit flags for Needle properties.
- **NameSize/Name**: 1 byte + variable, optional file name.
- **MimeSize/Mime**: 1 byte + variable, optional MIME type.
- **LastModified**: 5 bytes, optional last modified timestamp.
- **Ttl**: 2 bytes, optional time-to-live.
- **PairsSize/Pairs**: 2 bytes + variable, optional key-value pairs.
- **Checksum**: 4 bytes, CRC32 checksum of the Needle data.
- **Timestamp**: 8 bytes, append time (only in v3).
- **Padding**: 0-7 bytes, to align the total Needle size to 8 bytes.
---
## Version Comparison Table
| Field | v1 | v2 | v3 |
|---------------|----|----|----|
| Cookie | ✔ | ✔ | ✔ |
| Id | ✔ | ✔ | ✔ |
| Size | ✔ | ✔ | ✔ |
| DataSize | | ✔ | ✔ |
| Data | ✔ | ✔ | ✔ |
| Flags | | ✔ | ✔ |
| NameSize/Name | | ✔ | ✔ |
| MimeSize/Mime | | ✔ | ✔ |
| LastModified | | ✔ | ✔ |
| Ttl | | ✔ | ✔ |
| PairsSize/Pairs| | ✔ | ✔ |
| Checksum | ✔ | ✔ | ✔ |
| Timestamp | | | ✔ |
| Padding | ✔ | ✔ | ✔ |
---
## Flags Field Details
The `Flags` field (present in v2 and v3) is a bitmask that encodes several boolean properties of the Needle. Each bit has a specific meaning:
| Bit Value | Name | Meaning |
|-----------|-----------------------|--------------------------------------------------------------|
| 0x01 | FlagIsCompressed | Data is compressed (isCompressed) |
| 0x02 | FlagHasName | Name field is present (NameSize/Name) |
| 0x04 | FlagHasMime | Mime field is present (MimeSize/Mime) |
| 0x08 | FlagHasLastModifiedDate| LastModified field is present |
| 0x10 | FlagHasTtl | Ttl field is present |
| 0x20 | FlagHasPairs | Pairs field is present (PairsSize/Pairs) |
| 0x80 | FlagIsChunkManifest | Data is a chunk manifest (for large files) |
- If a flag is set, the corresponding field(s) will appear in the Needle layout at the appropriate position.
- The `Flags` field is always present in v2 and v3, immediately after the Data field.
## Optional Fields
- Fields marked as optional in the layout tables are only present if the corresponding flag in the `Flags` field is set (except for Name/Mime/Pairs, which also depend on their size fields being non-zero).
- The order of optional fields is fixed and matches the order of their flags.
## Special Notes
- **isCompressed**: If set, the Data field is compressed (typically using gzip). This is indicated by the lowest bit (0x01) in the Flags byte.
- **isChunkManifest**: If set, the Data field contains a manifest describing chunks of a large file, not raw file data.
- **All multi-byte fields** are stored in big-endian order.
- **Padding** is always added at the end to align the total Needle size to 8 bytes.
- **N = DataSize, M = NameSize, K = MimeSize, P = PairsSize** in the layout tables above.
For more details, see the implementation in the corresponding Go files in this directory.
|
