Demystifying NGE ROCK RIDGE

NEXT GENERATION SECRET STRAP1

Scope and Aims Ingest more events feeds as new accesses come online Increase maturity and availability of QFDs Pull through more QFDs based on Ops priority Deliver QFDs capable of holding ‘Convergence’ data and wider event types Provide a data mining and collaborative QFD development facility (BLACK HOLE - part of ROUGH DIAMOND) SECRET STRAP1 Enable sharing of QFD data with 2nd and 3rd Parties Interface with visualisation services in FIRE STORM

What is a QFD? Designed to answer single analytic question (e.g. ‘where is my target?’) Pioneered by ICTR, now developed by a community including Next Gen Events, ICTR, SD, GTE, … Simple table structure compared to traditional multi-function databases (e.g. HAUSTORIUM) No specialised database technologies so simpler to develop and maintain Question Focused Database Additional instances can easily be deployed at new locations or to increase capacity Smaller size and lower complexity means easier and quicker to develop and change SECRET STRAP1

What does each QFD answer? When was my target on line? Where was my target on line? Mutant Broth Who is my target interacting with on social networking sites? What web pages was my target looking at before going to this dodgy website? HRMap Who’s been visiting this dodgy websites? Karma Police Who’s been posting (vBulletin boards) to this forum? GooBzs (QFD Query Federator) Social Animal Marbled Gecko What part of the world has my target been looking at? Infinite Monkeys AutoAssoc What files have my target been uploading/downloading? Who’s been looking at this suspicious part of the world? What websites has my target visited? What alternative identifiers can I use to search for my target? What is my target doing online right now?! Samuel Pepys (Coming soon!) SECRET STRAP1 What posting (vBulletin boards) activity has my target been up to? Memory Hole Who’s been searching for these suspicious things on-line? What has my target been searching for on-line?

Ingest roadmap Internet Presence Events from 10G bearers eAD events Feb 10 SALAMANCA Telephony Events MONOPOLY Special Source Events Initial Converged Events from TERRAINS CDMA2000 test events Broadband RADIUS events Mar 10 Apr 10 Further 10G bearers at RPC1 (Bude) May 10 Jun 10 Trial part 1 - MUTANT BROTH, INFINITE MONKEYS, HRMAP, MEMORY HOLE from mobile tunnels Experiment Explore Deployed across CPC and RPC1 Trial part 2 - MMS, Blackberry, Google Maps, mobile Hotmail, mobile Gmail from mobile tunnels Explore Jul 10 Aug 10 TPS are working with the NGE Project and SMO Mobile theme to produce internet presence and application usage events from within mobile phone ‘tunnels’ in internet bearers. These will be trialed before full operational rollout Deployed across CPC and RPC1 Trial part 3 - Hotmail, Gmail, mail RU, Yahoo webmail from internet bearers Experiment Explore Deployed across CPC and RPC1 Trial part 4 - Windows Live IM, Yahoo Mail, SIP from internet bearers Explore Deployed across CPC and RPC1 SECRET STRAP1 ‘QFD style’ events will also be produced for types of event traditionally fed into the older HAUSTORIUM and HARBOUR PILOT databases

i 5 8 can. ww?wmu?s-r Screenshots from evolved MUTANT BROTH web interface, and an export of it?s data to Google Earth SECRET STRAP1 Convergence QFDs This major thread of work will: Store events where internet applications are accessed from a mobile device Allow to relate mobile device identifiers to internet identifiers such as email addresses Enable QFDs to store other more diverse event types, such as telephony events (currently SALAMANCA), and email events (currently HAUSTORIUM HARBOUR PILOT) Interface to LOOKING GLASS visualisation coming soon (in FIRE STORM work package) NEXT GENERATION events

SAMUEL PEPYS QFD Purpose: Provide a near-reaI-time diarisation of any IP address Search Term Search HuaL User-Agent Cuukir: [inn- - oar-)2; 1-3 01.102110 01.60310 .- 01.102110 1' Olf?lj?i? Expand all Normans-ad query Accept- tang Lla ge 00:0? 59 23:41:12 23:41:08 23:33:28 23:38:26 90.23? 90.233ll 90.237 90.233ll 90.237 - Collapse all A Export CSV Export raw -1 1 1'1 Results reports IP address ?-as (low con?dence), SE (medlum con?dence). Date Time Source Destination Type Description - 02(024'1'3 00:03:01 90.23?? 205.138.145.65 ?Jet-sea Visited {afterse Bearer GWUSCSOZI Connection TCP: 5143's to 205.1?8.145.65 port 30 ootswold cotswolcl prUHIL-uru (compatible; FISIE Gull; ?.rindows NT 5.1; Tridentfv'l?; .NET 2.0.5072i'; .HET CLR 3.5.303'29; CLR 3.0.303'29; CenLen PC 6.0; eSubiS-uba'riLI-gr 2.0.4.16] I I 205.136.145.65 205.138.145.65 105.178.145.65 IITFP 205.138.145.65 VuiLuLl 3? 6Flow[s} 205.178.145.65 Prototyped by ICTR Currently being pulled through by ROCK RIDGE, will be scaled to full 10G volumes by May 2010 NEXT GENERATION SECRET STRAP1 even?

BLACK HOLE What is BLACK A flat file store housing all data from a wide range of feeds (events and content) Provides a set of tools for accessing that data. Intended to be the source of events (and limited content) for the development of new QFDs and analytics. Contains a rolling 6 months retention Part of ROUGH DIAMOND What does it enable? New QFDs to be rapidly prototyped, then to be added to the operational QFD suite Trialling of new bulk analysis ideas New sources of data to be introduced quickly into existing QFDs. Users to look for particular patterns and behaviours (target discovery) TR, GTAC and GTE access to more data for research purposes, which may not be QFD related. NEXT GENERATION SECRET STRAP1 even?

User Feedback 'its amazing to see how the pace of delivery in TDB has increased and have been impressed by your responsiveness to customer needs.? (_,Senior User) 'Absolutely FABULOUS well done (Iain Lobban, ref SUPERDRAKE reporting) ?Almost exactly a year ago I set you the challenge of delivering an upscaled massive events capability in order to support Internet Operations being conducted by GCHQ. Through your stripy team working on BLAZING SADDLES, BLUESHIFT and SUPPORTING INO you successfully met this challenge and delivered us a significant new capability in July.? Deputy Director Cyber Operations) 'Bloody awesome' (analyst, ref SUPERDRAKE QFD) ?lt's working flawlessly' (analyst, ref BLACK HOLE) NEXT GENERATION SECRET STRAP1 even?