Documents
QFD BLACKHOLE Technology Behind INOC
Sep. 25, 2015
QFDS and BLACHOLE
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to
Slide 1
Technology behind
1:
=1
GCHQ on
QFDS and BLACHOLE
This information is exempt under the Freedom of Information Act 2000 (FOIA) and may be exempt under other UK information legislation. Refer any FOIA queries to
Slide 1
Technology behind
1:
=1
GCHQ on
Probes Flat Store Analytic engines
1OG
Multi
Clouds
Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 2
Probes Flat Store Analytic engines
1OG
Multi
Clouds
Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 2
BLACK HOLE
- Flat data store not a database
I
- 1,100,000,000,000 events since 6th August 2007
00
53.3TB compressed 12/ lSthiearC
217TB uncompressed 'nggjivemg
47% of data is from last 3 months Isocia'networking
19% Instant Messenger
I.Routing Protocols
- ~10 Billion events per day
- Cheap system:
?1 per TB of storage
?20k per 10Gbps probe
~?400k for 10x10Gbps prototype
Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of limb.
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
4 J6 subject to exemption under
Slide 3
BLACK HOLE
- Flat data store not a database
I
- 1,100,000,000,000 events since 6th August 2007
00
53.3TB compressed 12/ lSthiearC
217TB uncompressed 'nggjivemg
47% of data is from last 3 months Isocia'networking
19% Instant Messenger
I.Routing Protocols
- ~10 Billion events per day
- Cheap system:
?1 per TB of storage
?20k per 10Gbps probe
~?400k for 10x10Gbps prototype
Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of limb.
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
4 J6 subject to exemption under
Slide 3
Questions INOC Today
Who, Where, When, What? Online presence (TDI)
- MUTANT BROTH
Who, which website(s)? Online browsing activity
- KARMA POLICE
Who, which fora?? Bulletin board usage
-
INFINITE MONKEYS
Where? Google maps/earth usage
Crown Copyrig - eserM?Is?l?nl'?o?sQemp under the Freedom of Information Act 2000 and may be subject to exemption under
- - formation legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 4
Questions INOC Today
Who, Where, When, What? Online presence (TDI)
- MUTANT BROTH
Who, which website(s)? Online browsing activity
- KARMA POLICE
Who, which fora?? Bulletin board usage
-
INFINITE MONKEYS
Where? Google maps/earth usage
Crown Copyrig - eserM?Is?l?nl'?o?sQemp under the Freedom of Information Act 2000 and may be subject to exemption under
- - formation legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 4
Questions INOC Today
Who, Where, When, What? Online presence (TDI)
- MUTANT BROTH
Who, which website(s)? Online browsing activity
- KARMA POLICE
Who, which fora?? Bulletin board usage
-
INFINITE MONKEYS
Where? Google maps/earth usage
- And many more
Crown Copyrig - eserM?Is?l?nl'?o?sQemp under the Freedom of Information Act 2000 and may be subject to exemption under '7
- formation legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 5
Questions INOC Today
Who, Where, When, What? Online presence (TDI)
- MUTANT BROTH
Who, which website(s)? Online browsing activity
- KARMA POLICE
Who, which fora?? Bulletin board usage
-
INFINITE MONKEYS
Where? Google maps/earth usage
- And many more
Crown Copyrig - eserM?Is?l?nl'?o?sQemp under the Freedom of Information Act 2000 and may be subject to exemption under '7
- formation legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 5
Questions INOC Today
Too many specific Too much for to learn.
Need analyst systems to be simple and intuitive.
Let find simply understand their targets
drill in to systems only when they need to
Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 6
Questions INOC Today
Too many specific Too much for to learn.
Need analyst systems to be simple and intuitive.
Let find simply understand their targets
drill in to systems only when they need to
Crown Copyright. All rights reserved. This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 6
AR Prototypes - EPR Stats - Feb 2008 - Jan 2009
End Product Reports contributed byCrown Copyright. All rights reserved. This Information IS exempt from disclosure under the Freedom of Information Act 2000 and me be subJe to exemption under
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 8
AR Prototypes - EPR Stats - Feb 2008 - Jan 2009
End Product Reports contributed byCrown Copyright. All rights reserved. This Information IS exempt from disclosure under the Freedom of Information Act 2000 and me be subJe to exemption under
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 8
Number of users who executed 10 queries, by area
Number of users
000? ?23 0%9gx
<30 Qo?ooerQOQ
09?
<26<
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 9
Number of users who executed 10 queries, by area
Number of users
000? ?23 0%9gx
<30 Qo?ooerQOQ
09?
<26<
other UK information legislation. Refer disclosure requests to GCHQ on
Contains Intellectual Property owned and/or managed by GCHQ. The material may be disseminated throughout the recipient organisation, but GCHQ permission must
be obtained for dissemination outside the organisation.
Slide 9