Documents
Data Stored in BLACK HOLE
Sep. 25, 2015
TOP SECRET STRAP1 COMINT
The maximum classification allowed on GCWiki is TOP SECRET STRAP1 COMINT. Click to report
inappropriate content.
For GCWiki help contact: webteam
. Support page
Data stored in BLACK HOLE
From GCWiki
Jump to: navigation, search
This page describes the data which is stored in
BLACK HOLE, where it comes from and how
much of it there is. Some of the information is
too highly classified to be contained here, or
falls under the Applied Research intellectual
property restrictions. Nonetheless, as much as
possible is published here.
Contents
1 What data BLACK HOLE contains
1.1 Types of data
1.2 Classification of the data
2 Where the data in BLACK HOLE
comes from
3 How much data there is
3.1 Volumes in BLACK HOLE
3.2 Volumes in the datamarts
[edit] What data BLACK
HOLE contains
[edit] Types of data
The events created cover webmail, email
transfers, ftp, chat, internet browsing, website
logins, vbulletin web fora, web cams, gaming,
social networking -- and the list is growing.
BLACK HOLE: The Knowledge
What it is
Where it came from
What is in it
What data it contains
How much data there is
How to use it
How to connect to it
How to find the file you want
Who uses it, and why
Applied Reseach uses
SIGINT Development uses
IPT uses
How it is built
Logical connectivity
Physical system
Security issues
What could happen to it
Applied Research enhancements
SIGINT Development intentions
Engineering interest
What we like about it
Good things
Things that could be better
Exploding myths
Wish list
v·d·e
The data is primarily generated by B13B high-rate bearer processing modules, and so the exact types
depend on which modules are running at the time.
[edit] Classification of the data
The classification of the data in BLACK HOLE ranges from SECRET STRAP1 to TOP SECRET
STRAP2 UK EYES ONLY. Most data should be covered by TOP SECRET STRAP2 and there may,
TOP SECRET STRAP1 COMINT
The maximum classification allowed on GCWiki is TOP SECRET STRAP1 COMINT. Click to report
inappropriate content.
For GCWiki help contact: webteam
. Support page
Data stored in BLACK HOLE
From GCWiki
Jump to: navigation, search
This page describes the data which is stored in
BLACK HOLE, where it comes from and how
much of it there is. Some of the information is
too highly classified to be contained here, or
falls under the Applied Research intellectual
property restrictions. Nonetheless, as much as
possible is published here.
Contents
1 What data BLACK HOLE contains
1.1 Types of data
1.2 Classification of the data
2 Where the data in BLACK HOLE
comes from
3 How much data there is
3.1 Volumes in BLACK HOLE
3.2 Volumes in the datamarts
[edit] What data BLACK
HOLE contains
[edit] Types of data
The events created cover webmail, email
transfers, ftp, chat, internet browsing, website
logins, vbulletin web fora, web cams, gaming,
social networking -- and the list is growing.
BLACK HOLE: The Knowledge
What it is
Where it came from
What is in it
What data it contains
How much data there is
How to use it
How to connect to it
How to find the file you want
Who uses it, and why
Applied Reseach uses
SIGINT Development uses
IPT uses
How it is built
Logical connectivity
Physical system
Security issues
What could happen to it
Applied Research enhancements
SIGINT Development intentions
Engineering interest
What we like about it
Good things
Things that could be better
Exploding myths
Wish list
v·d·e
The data is primarily generated by B13B high-rate bearer processing modules, and so the exact types
depend on which modules are running at the time.
[edit] Classification of the data
The classification of the data in BLACK HOLE ranges from SECRET STRAP1 to TOP SECRET
STRAP2 UK EYES ONLY. Most data should be covered by TOP SECRET STRAP2 and there may,
inadvertently, be ECI data (but this is not expected, as we are not actively doing things covered by ECI).
The data includes fields which have been produced by enrichment (such as, for example, target details)
and this can increase the classification.
[edit] Where the data in BLACK HOLE comes from
Take a look under the covers of BLACK HOLE to find out where the data comes from.
[edit] How much data there is
[edit] Volumes in BLACK HOLE
The total amount of (compressed) data entering BLACK HOLE is approximately 200GB/day.
[edit] Volumes in the datamarts
Prototype
Current
retention
time /
months
Current
database
size / TB
Estimated size
for six months
retention / TB
Estimated
Current
database size for 6
database size
months retention /
/ billion rows
billion rows
MUTANT
BROTH
4
7.7
11.55
21.7
32.55
KARMA
POLICE
3
6.8
13.6
17.8
35.6
MEMORY
HOLE
0.5
0.6
7.2
2.2
26.4
AUTOASSOC
6
0.1
0.1
11
11
SOCIAL
ANIMAL
1
1.6
9.6
1
6
HRMAP
3
7
14
16
32
INFINITE
MONKEYS
6
0.02
0.02
0.2
0.2
143.75
Total
23.82
56.07
69.9
Please note - all of these values are approximate.
POC:
Retrieved from
Category: ICTR BLACK HOLE
Data_stored_in_BLACK_HOLE"
(mail Phonebook Find)
inadvertently, be ECI data (but this is not expected, as we are not actively doing things covered by ECI).
The data includes fields which have been produced by enrichment (such as, for example, target details)
and this can increase the classification.
[edit] Where the data in BLACK HOLE comes from
Take a look under the covers of BLACK HOLE to find out where the data comes from.
[edit] How much data there is
[edit] Volumes in BLACK HOLE
The total amount of (compressed) data entering BLACK HOLE is approximately 200GB/day.
[edit] Volumes in the datamarts
Prototype
Current
retention
time /
months
Current
database
size / TB
Estimated size
for six months
retention / TB
Estimated
Current
database size for 6
database size
months retention /
/ billion rows
billion rows
MUTANT
BROTH
4
7.7
11.55
21.7
32.55
KARMA
POLICE
3
6.8
13.6
17.8
35.6
MEMORY
HOLE
0.5
0.6
7.2
2.2
26.4
AUTOASSOC
6
0.1
0.1
11
11
SOCIAL
ANIMAL
1
1.6
9.6
1
6
HRMAP
3
7
14
16
32
INFINITE
MONKEYS
6
0.02
0.02
0.2
0.2
143.75
Total
23.82
56.07
69.9
Please note - all of these values are approximate.
POC:
Retrieved from
Category: ICTR BLACK HOLE
Data_stored_in_BLACK_HOLE"
(mail Phonebook Find)