XKS System Administration

TGIF SECRET SI REL TD LISA, AUE, CAN, GERill-EFF - - I .. . - I. . p?u?qdyste rri December 2012 I A. -l-u H,

lirali'IL I: TUF EECRET 3 REL TU USA, AUS, CAN, GER, MEL 2

@339 Lesson bjectives Vintroduction to XKEYSCORE VPurpose and Capabilities VData Flow VWhat is a Cluster? VXKEYSCORE Databases TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 3

it" El] fill EEL WED MBA, ?31339 ER, NZL introduction . XKEYSCORE performs filtering and selection to enable to quickly find information they need based on what they already know. . XKEYSCORE also performs SIGDEV functions such as target development to allow to discover new sources of information. TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

atria til" at a Eli to) USA, ER: . 1-.I . .. 4 Fun-1 i 5 .xln'trod cti 0 . XKEYSCORE processes data at field sites, where it is collected, and allows from all over the world to query it. I At field sites, the XKEYSCORE software can run in clusters of few or many servers, giving it the ability to scale in both processing power and storage. . All processing is plugin or fingerprint based, which allows new capabilities to be quickly deployed to support operational needs. EECRET Si REL TU USA, AUS, CAN, GER, MEL 5

a] a IEEJL. The use. awe. BAN. ER. NZL capabiliti'e - Purpose and . XKEYSCORE is a Computer to Computer (020) exploitation system. I It is a fully distributed processing and query system. . XKEYSCORE can run on multiple servers. . Plugin and fingerprint architecture allows new capabilities to be quickly deployed. TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

Egl?tf??l?il?'ff lit" a] a mat. mamas AUS?r'dwa re . XKEYSCORE is typically installed with Red Hat A85u8 operating system. The suggested disk set up ISZ . Set up separate partitions for/ (root), /var. ltmp, and lexport/data . XKEYSCORE clusters can be composed of three different functionalities, which are: 0 One host acts as the web server/user interface, 0 Another host normally runs as the real-time processing unit Other host acts as the search or query system. . Hybrid system can perform multiple roles on one server, which enables efficient registration. 0 process_data_parent 1 query_proc EECRET Si REL To USA, AUS, CAN, GER, MEL 7

t??ig??itl. @1339 I Data Flow (High-Ie el) The backend is where the raw data for XKEYSCORE is processed; that is, we receive information from our sources process it, and store it into a database. a. engine] (ussr queries) ans num ers- 1-- mstadsts tables 9 email addresses 4? lug ins TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 8

. r? ran it Esta; [@559 mg ER, NZ Data Flow - Cluster}; . A cluster is Comprised of one master server and one or more slaves. . All slaves in a cluster have their own copy of configurations (/opt/xkeysoore/oonfig) files via the push_config cronjob. TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

'?liilill tut-?it, @3339 Data Flow - Databases There are two types of databases on XKEYSCORE system: insert (i0) and query ((310) D124 Gasman; _?ir - I 12': I I r" n- I- 3cm: in - ragisLar_rn-Ertadata_tablas sassluns I 4 .sntLinputjrm -. Ea NOTE: sotf_input_proc is now called, sotf_dist process_dataN?s are now called, precess_data_parent TUF EECRET rr 3 REL TU USA, AUS, CAN, GER, NZL 10

?f'ti-EJE' ma NSF, AUS: BAN: GER: - Databas . file_input_proc and sotf_dist take in sessions from the front-end and load balances them across multiple process_data_parent?s. . process_data_parent is responsible for processing sessions and extracting metadata . xks_meta_ingester takes the metadata from the process_data_parent?s and writes it to the insert database, i0 - register_metadata_tables takes completed insert tables, indexes them, and moves them to the query database, qO TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 1 1

. ll . - TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 12

:1 JEFJHEH @3139 Lesson bjectives Operating System Services NFS Mount Points V/st_data Directory Structure TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL l3

:S-Ell It? IEEJEIL. Tr@ 493% @9311, FL NZL tem Service perating . XKEYSCORE is typically installed on servers running Red Hat 5u8 operating system. . This section discusses common operating system services used during XKEYSCORE operation. TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

it: $31317 51'? Eli-?Til Jilin." Trle 45m @2531: II I The daemon is needed for the web-based GUI, viewing content, and is required on all servers. . The master server is the web server and the slaves retrieve Content through TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 15

:asis? 932le it as; use, 52MB, @1111, NZ . The daemon is a SQL-based database server for processing, querying, and is needed for the XKEYSCORE GUI. . It is required on all servers for administration, processing, and querying metadata in databases. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 16

it: a] it? Eli. Thai) @3339 age, ER. NZL ii I. . Mounting a directory uses the NFS service. . NFS allows file systems that physically reside on one computer to be shared by other computers on the network. I The machine with the hardware containing the directory must allow the hardware to be made available to other machines. . Required on all computers for clustering. TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 1?

. $2121; 333% . late/exports lexport/data/xkeyscore master(rw) slave(rw) TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 18

vim aeraa??' til" en a mat. mam AUSComputers requiring shared access to the /export/data/xkeyscore directory must be told where to find the directory. 0 This is accomplished via automounting. . The autofs daemon listens for computers trying to connect to the directories, or mounts, that it is responsible for. The mounts are dropped after a time out, but autofs remounts the drive when drives need to be accessed. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 1?3

vigil 'E'afElEiif??lEHE'?' it: lit FEEL. treat AUSclustered XKEYSCORE, automounts must be set up on all of the computers in the cluster. . automaster and auto.data files in the /etc directory must be edited or created. . When finished, the mounted directories on the remote machines can be accessed. . The oper account should have full read/write permissions on all shared drives. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 20

it" til] [3313; ?3.33, ER, NZL Mo'unt Points . automaster? designates mount points on the local computer and the directory to mount on the remote server. 0 Example: kas_data Ietc/autodata --timeout=60 . auto.data enables all servers to see the /export/data/xkeyscore directory on other machines and locate databases, archived, data, and MAILORDER directory. 0 Example: xks1 -rw,soft,intr,tcp xks1:Iexport/data/xkeyscore ?rw,soft,intr,tcp xksZ:lexport/data/xkeyscore EECRET Si REL To USA, AUS, CAN, GER, NZL 21

was rm NSF. AUS: BAN: GER: ;Hir?ctory Structu re :7 . contains all of the XKEYSCORE software. Software includes the GUI, processing, scripts, and configurations. . bashrc XKEYSCORE environment variables file. 0 beacon] - contains the beacon perl script (shm_beacon.pl) and a link to the beacon configuration file . bin.she lsl and - contains miscellaneous bash, python, and shell scripts. 0 build! - contains libraries and plug-ins. 0 install] - contains installation scripts. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 22

ry StrUCtu re .th . consists of sub-directories 'if'tLillE' lira NSF, AUS: BAN: GER: and each contain configuration files for building and running XKEYSCORE. crontab] contains the master and slave crontab file. dictionaries! - contains the dictionary files for the filtering, selection, TRAFFICTHIEF, CADENCE, fist tables, and any other local dictionaries. miscl - contains miscellaneous per?plug?in configuration files, sotf_input_proc.xml plugins/ contains event handler configuration files for each of the plugins - contains web configuration files and xscorecfg. - contains the config files for all the services needed by XKEYSCORE php, etc.) 23 TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

it: at] it? ?e urea, AUS, BAH, ER, NZL ,-.irectory Structure FA I - - contains the contents of the web front end. 0 docs! - contains documents viewable through the XKS GUI. . - contains web pages and scripts that are not on the secure server. 0 secured] - contains web pages and scripts that are on the secure server including: cronsI - location of cron job scripts - contains source code for the XKS GUI. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 24

etiijziit?ia??' it: an ill lid-EL to) sea, AUS, BAH, ER, NZL girectory Structure FA . lexportldatalxkeyscorel - is used for both internal databases and metadata archive databases, input, output, and archiving of data. . archives! - (optional) destination for processed content 0 inputs] - (optional) used for file based input - location of the database consisting of admin, insert, and query databases. 0 outputs! - (optional) contain the following sub-directories: mailorderl - pickup point mailorder_workingl - file creation point before being moved to mailorderf EECRET Si REL To USA, AUS, CAN, GER, NZL 25

a all Eli. TIE) [relate aye, ER, NZL Directory Structure . kas_datal - logical mount point for all other XKEYSCORE (including itself) /export/data/xkeyscore. - <hostname>l - mount point for the hostname?s local directory /export/data/xkeyscore (referenced by host name). All servers must export their /export/data/xkeyscore directory and mount this on the /<hostname> directory for each hostname of each machine, including itself. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 26

lirali'IL TUF EECRET 3 REL TU USA, AUS, CAN, GER, MEL 2?

r-L ta: as]; 17115;] @3339 GEL i Lesson bjectives Accessing the GUI VExiting a Session Main Menu Bar Admin VComputer Resources Option VStart and Stop Processing VRun a Process Manually Users Search Workflow Central Results Fingerprints TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 28

atajzti??fa??' it: all ?e meal, AUS, BAH, ER, NZL ACCessing the GUI . In the address field of a web browser, type hosfname or IP address>. . PKl?s or a UserlD and password are required. After successfully launching a new session, the XKEYSCORE WELCOME window appears. 0 Note: Compatible web browsers for XKEYSCORE version 1.5 are: Internet Explorer is not supported Firefoxi3.0.* and above EECRET Si REL TU USA, AUS, CAN, GER, NZL 2?3

. I: ll'll . 72:12] Accessing theGUI ?ims- I?Wirlf': ?fal?nr iLzer: :L'yaerzl' ?'n'Jd?muLa'tral ,I-?jaJt; I-rgarprr: art-353:: ?lial: Eli-Hz? ii . {?71 km: 2 I35 .55: hart": nap-E: t' erthaTWHEtLI IE r: [Ti-f: m: EH ?Jag: ,1 Emma mmuw nu: L-?l ?czia'ty Elm- 113311?: I Hislunran EMF Irr. Elm 1cm Enc?e ?El-hdmr MEIHUHEE Lnarnl-Inw SSH-macaw; E. Emu): Flax-J13: .I'th Uri-zines a El? HUMAN HIE-HTS ACT. 5:33-35 ussm 13 AND 1155.": 9 a - - F: All [memes naqume In: 352.155 F'ij?ui Ad UHHIH 1n E. [It-:45: turlliarlnl. iriJrrIIaliJrI Rabid LI:er Ha: as grumple hull 1119 mm imar?cn. An audit um? 11::th hum mmlimcd and Hi tut: mamhm? 1E5J_m.rm3 Ba pa?pl Hemlh Htatmn's respmsetn 311' tn (m I I ample-1t hmu?Tt under IRA and a5 part uflhe If: Shh-F" Fm? F'Inaar. ?rha?r E421 J?rs i5 ruqu'ud [m bqurL- Ema- whinillinn qunrr "null-dual; terms wmi?c In a mummy name. Emmy: Hunter]. aunh as Elam FHH. addre35. m?pmtl'hmk Hm FITI is ?nd FIF. t_ I: In." rt 3 Elrili'jh THnilurf ur '3 mm Samuel Parl'r 'pnaraann' urlh] in I??t?d i1 1119 UK. EDT or Second murlrj. ETA i5 Elan raqmrerl ?rurmlduard are Imahlj' 1: I I I [pirg 111 retrial. a slilslarlial LE- Hum? EI1FEF. nn 3 UH Li] Ernie} FII a: 1' cl h: legil gLiiHnL'L- milaIJIE hurl '17: hr lecr 51w; Statinn. njl; El TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 30

. . a? {sit 33%;le till W2) [Em ?31339 EFL NZL Main Menu ptlons 1, . The main menu bar across the top of the window has menus that, when selected, each has additional options available in a drop down menu form. ?Hnme ?ndmin Mars Search ,Results Eiingerprints Eiaggimg Statistitrs flailing @llap TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 3 1

Main Men Home ptions . Returnto the main page. to esstasis? it: fell ill last; as sea, ?11m 6AM, EFL NZ .t Can edit user settings, disablefenable access to databases, edit a search form search setting, and restore default settings. Admin Computer resources, Input Directories, Category Throttle, Search DBs, and DB Registration settings. Users Contains UserAccounts, Clearances, Privileges, Send Email, Users Online, My Auditees, My Audit Logs, and All Audit Logs. Search Provides different search query forms, such as email addresses, category, full log, and user activity. Workflovv Central Request modify, and vievv standing queries that will execute at a specified time or interval. Results Can search personal searches by date time, query type, query name, output table, and user. Fingerprints Fingerprint builder and vievver. Map Brings up Google Earth Help Help Documentation, XK Forum, Account Maintenance, and About XKEYSCORE rr Si REL To USA, AUS, can, GER, NZL 32

Admin Menu - to: @3339 I i Computer Resources Allows for process configuration and management. Input Directories Contains the configuration for file-based input directories. Category Throttle Edit CADENCE quota limits by category and/or fist table. Search DBs Configuration for query databases which are queried when a search is submitted. DB Registration Contains the mapping from insert databases to query database. News Add, modify, delete mandatory and home page News. EECRET Si REL To USA, AUS, CAN, GER, NZL 33

'i'i?ElE?tfrEJEilE'Tr' a all it? to) area, AUS, BAH, ER, NZL ornputer Resou rces The Processing->Computer Resources option from the ADMIN menu allows control of the entire daemon-styled, or continuously running, processes for XKEYSCORE. . Processes appears in a table following the conven?on: xkeyO?l process_data_parent EECRET Si REL TU USA, AUS, CAN, GER, NZL 34

. .. . II -. .. .521 ll: Jiilu'E-Sg @1339 Computer Resources Cnmputer Resource Windnw Praness Table Home I MWKS Fadnin i Users CL Eva-a'cl'i Results 1.1; Finglarprirll? i-alistics @l'llap Heb IHEIIJ "l Navigation Fltlar El Prncesshg Camputer Flasaurces Inau: Diracmrias ET I: .-I EIJHHIHG -: I: Fl. I: STD HE F: 15713an Cats-gnaw Thrattls I: ?El Databases Beach DE: DE Registration 1: Cl L :ili :ias Casenntatisn Blacklist Er] Reload Carl?in: Files [El Haws 1!le Ip Summary Tabla 11'le Crashlnuqar 11'le Startup mars! PrnFilar In computer Help Add Austin-la 1' App is Running I simian: Pris: Has?: Pragrn?n Marni ?rmed: Prawn-Irr- Pt- Emil-pried Ei?u: Slain: Date?mir aim-tea Damn-ii Stunned . tlxiraw?'i GUM 21312-1 2-03 15:11:13.0 . tlxiraw?i quaryjmi: m5 21112-1141? 1?:31510 Eli-1 2-11-21!r 11132-110 . tlxiraw?i 311115 2M2-11-H1E3351EI 2312-11-2? . tlximrr?i :sisaJnsiaJngasta' 31051 2912-11-2? . ?lm-Estranme 31m Fl.? 201 2-11-21 . tlximw?'i quary?lspa'im 1311 21112-1303 . tlmr??i filiJ?pL?jT-??: 311 I34 2-11-21 . tlxisaw?i Elsaj's'st?m?wr?w 1% El.? 2012-11-2? 21:01:13.9 251 2-11-21? 21:31:13.0 . tlxiiaw?i sat?adi?zsiaawar 311 IJB Fl.? 2812-11-2? 1?:32h1? .EI . tlxiiaw?i 311 21 2312-11-2? 11132410 . tlxlia'w?i nadenlaBJ?Sl-?ngj?c nil-dug "w I5 311 2312-11-2? 1?:3231? .EI . tlxliaw??i 311 33 21312-11-2? ??3510 . tlailisw?i mallarderjmc 311 ill) Hi.? 2312-11-2? 1?:3231? - Hyman-01 "In-5mm 311 43 21312-11-2? 2012-11-2? TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 35

ill" a] a mag mam AUS: tie?: ?3 BE ,Computer ResourceSEil 1 I . The xks_app_launcher process runs on all servers from the inittab. . It tells the computer which program to run by looking at its tasking host. - The config file specifying the location of the tasking database. . Processes can be stopped, started, edited, or deleted from the Computer Resources window. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 36

e] ll Eli. The [Em lama, BAN, ER, NZL Computer Resources - . Add a new process click Add - Edit a process click Stop in the ACTION column, then click Edit. . Delete process click Stop in the ACTION column, then click Delete. - Stop the App Launcher disables the xks_app_launcher on every host. TUF EECRET 3 REL TU USA, AUS, CAN, GER, MEL 3?

a at at? REL its) use, AUE, ER: ,Rfeso urces Colo nventio . Visual cues in the form of colors are used to help identify activities performed by XKEYSCORE and serve as status indicators for monitoring purposes. 0 Red indicates processes have been stopped 0 Green indicates processes are running . Yellow indicates processes are starting 0 Orange indicates processes are being stopped 0 White indicates processes won?t start . Visual cues are also available in the COMMANDED STATUS and STATUS columns of the table. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 38

a er] a REL tie} USA. AUE, ER: Ul?Ce s?Sta l?t/ tf? . It may be necessary to stop or start processes for troubleshooting or for a graceful server restart. . Individual processes and programs Click Stop in the ACTION column. To start it, click Run. . To stop all individual programs, select Resources. Enter the program name in PROGRAMS field, then click OK. I Can use ?xks proc? actions and commands to do the same function TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 3?3

a a] at? REL tie} NEVA, AUE, ER: Ul?Ce s?Sta l?t/ tf? I All Processing - select Resources from the ACTIONS drop?down menu, leave the PROGRAMS and ON HOSTS fields to their defaults, click OK. - Specifying programs or hosts select STOP or START, enter a wildcard expression such as or in the PROGRAMS or HOSTS field, and click OK. 0 Example: process* . Alternatively, in a terminal window can run: 0 proc stop process* TUF SECRET 3 ii REL TD USA, AUS, CAN, GER, NZL 40

as it: a] it? so was, AUS. BAH. ER, NZL ,Rua Process Manual-l . It may be necessary to run a process manually for troubleshooting purposes. To run a process manually: 1. Launch the GUI and log on as oper or admin. 2. Click ADMIN Processing Computer Resources 3. Click Stop in the ACTION column for the process. 4. Open a terminal window and to the host running the process, as the user ?oper?. 5. Type p8 -ef grep <process name> to verify that the process is stopped. 6. Type <program name><program argument} . Example: guery_proo <program arguments, if any> ??logleve1 debug EECRET Si REL To USA, AUS, CAN, GER, NZL 41

133', 5-3: @339 Users Menu ?Hume Emma admin 4-: Fllte: .H Llsar' account's; El Clearancaa Privilege; El Send Email Users Dnline: . This menu is only accessible to users with system administration privileges. . An SA can add/modify user accounts, add groups, clearance levels, privileges, and email users from this menu. EECRET 3 ll REL To USA, AUS, CAN, GER, NZL 42

.. ?3534! eat [Elk-535% @339 ms . From the main menu bar, click to View your profile, accesses, privileges, auditors, settings, fingerprints, workflows, and recent results. . Right click on any search form name to add a shortcut for that search form. 11-: ff? . Hesigatien Filter it El Full Leg em HTTP Full Leg DHI ?ttisittr Fingerprints iris Werk?ews My Flecent F're?le Ms Fingerprints Results My My Ftecent Results F'reiile TUF EECRET rr Si REL TU USA, AUS, CAN, Gee, NZL 43

?i'ii-Zi} E, li?fi'?i?. (Ell: BEL. Search Menu I From the main menu bar, click SEARCH. Menu options display in the vertical pane on thele?. Fame: Wizard 111i mm Audra-.5 Eur Ina-pi 1hr Fun-LII:ch 'Ezu Salim-arc.- Us-cr 5mm? web Anmywizhs El:th :niJ Ill-Far!qu Fable- Ila"? F5 Stat-En Crud; w?mm Ilm WWI-ELISE Harms lanai Pianist-m Map: CINE I'Luq-n 'i'E E'I'uu l'r'lai Lung: Emall indies-:25 um Endpoint enact-ed Hardin: Endpoint Fit: EMF-?int Helen-ad fit-at: Endpeir?: Flam-shin: Ends-mint 5m: Endp-?in! Rented Cam-its Emir cu:th FIE Emu Mal Laughs Faith.- Faresiws- Mal:- Ful Lug H'l amass: Etuz?ms IE {ml-i=5 lie-5pm Ilsa a? a? 3? it- Eta-r Loggers Ht?bm 51' MP ham-55 InEmI'na-ti-an Ir-Enrmantien fide Mmmaihn Merit-3n PEP Chiral-liaisnilti Prism-?5 In Re?guy -j i? $151333Unr?ml?t Ream-14 E?amaiz-F- lit-urn incur InF-?u?aa-Eim thl'l'h?IlJi Web Server: Wind-1w: Lijm' EdsnliFim' vanish. Known Fania-curd Amt-mall tin-gs Ell-El Inna: Emil anl: M52 ME may: and LIME Lilith-Fit]! Fault-:1 Its-uniting FLJIEIH: Erna-m Pa?mn-rc? Llyur Acti'nr't'y' and Dmiit-nnl: avurcmi-cia-s "(eman- Milli-?IE: Ennis-rial: Eating-Ml; and In?l-Entt?d I?ll-H; Earner-SIS and UEIJIEILHTI Susan-n: TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 44

. When choosing a plugin type from the men?' Search Menu La: @3539 options, the only data searched is the data that was identified as a hit when the plugin was Category DNI urocessed. Searches dictionary category hits. Full Log DNI Searches all sessions received by XKEYSCORE. User Activity Enables a user to search by a user's activity. Example: 3 user can find a hotmail user?s msnMailToken EECRET Si REL To USA, AUS, CAN, GER, NZL 45

'ii'tirjgi?i' a all a Eli up) USA, ER: Search Menu Til 5 . All searches are conducted on database tables where the results of the XKEYSCORE engine are stored. . Each row of a database table contains values from an individual session that was identified as a hit by XKEYSCORE when that plugin or microplugin processed the session. . Each search type query is related to a plugin or microplugin, which performs the metadata extraction. TUF EECRET Si REL TD USA, AUS, CAN, GER, NZL 4.5

a @il a Eli to) USA, ER: Deta . Search details can be accessed from the Search status window by clicking Details. - CURRENT SEARCH DETAILS window displays and allows the user to watch a query run through the appropriate databases. . RESULTS link in the main menu bar can be used to display a list of all previous search results. . Queries operate in parallel on each host. TUF EECRET Si REL TD USA, AUS, CAN, GER, NZL 47

Details Windnw Query ?Fur?D4391 Pause- Shaw IEI I25 I32- I34- IL Eda-{new .?tl?li??r?I?l?IEI mum-3W r' I ??umm?rh?"F?cm JI'I?uI'I?unl'yJ'l?i?w I CI: 1 - .?tlh?il??H?"I?q 1 2: 5 2 .?tlh?ili?H?"I?q E: QUEFLHIHIEI r_ l .?tlh?il?iEH??F'I T: ?EummEFVmaw I 3-3-73 1345555 I .?tlh?il?iEH??F'I a: chL-Immar?wm-E-W 1 'El- .?tlh?il?iEH??I?ED: ED- .?tlh?ili?'?v??I?qu :cqaummal-?wm-aw TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 48

vigil it: lit FEEL. urea AUS: EFL .1 "ll. . I I . From the main menu bar, click RESULTS to retrieve the results of previous queries. . By changing the start and stop dates, queries performed between those dates can be viewed. . If the query name is known, it can be entered in the field. I If the USERID is known, it can be entered. . When complete. a window displays with the matching queries. TUF EECRET Si REL TD USA, AUS, CAN, GER, NZL 49

Ir: .. IMEIL TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 50

EJEIL ?31!ng @337 Lesson bjec?tiVe XKEYSCORE Process Data Flow Processing Programs Query Processes Other Processes Cronjobs Vcrontab TUF EECRET 3 REL TU USA, AUS, CAN, GER, MEL 5 1

iaaE'? THE $35-39 @313 EFL. NZL cess Data lilg) Back-End SOTF WU sntf_dist System and nther fnrma E?mp? ?rm: Scans dirs far new files I I data_parent) Insert db inutile handler queryglispatch - - Master TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 52

slew? sill it? @3121; Tr@ [Em 53% @5311: FL PrOCessing Programs - Processing programs are the main processes that extract metaata from the traffic and then dataase the information in insert databases. file_input_proc Scans for new input files. (before processing moves the file to the .tmp directory of the input directory specified) sctf_dist Listens for incoming SOTF sessions process_data_parent Processes all new files discovered by file_input_proc or sotf_dist; optionally archives content and databases metadata. Parent process loads all dictionaries and starts up, then forks child processes which do the actual processing. EECRET rr Si REL To USA, AUS, can, GER, NZL 53

as; gasses a a] a was tosses AUS: ?3 BE r0 Ge 8 ll": _d Processing - This process replaces process_dataO through process_dataX - The ?parent? process starts up and loads all the dictionaries, and then ?forks? child processes which actually do the processing - Parent acts similar to the xks_app_launcher, managing restarts forthe children when they die I When dictionaries are modified, parent reloads them and restarts the children - ?xks proc? will show an number next to process_data_parent . This is the number of children currently running, over the number that should be running (based on the xks.config num_data_processors setting) - will show up yellow anytime and green when everything is running normally - This means when you first (re)start pdp, it will show yellow while it is loading the dictionaries, because none of the actual child process_data?s are running yet . ?xks proc? will report extra or missing process_dataX with a PID of 0 .- Can?t tell what PID missing process_data is suppose to have, because its managed by the parent now TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

2.231% it? fzi-?Ell '1?er} {Elia}, @133 @3339 NL- uery Processes ii if i - Query processes are processes that seach and submit all necessary tables for the queries. query_dispatch Submits search jobs to search databases and propagates the status of the search and results back to the web server query_proc Searches through all the necessary tables for the queries. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 55

a an it? Eta; Wu) [@333 ER, MEL 3 1 . ther Proces . Other process which is run from the Application Launcher. - mailorder_proc polls the directory by default. Then renames and moves mailorder files to TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 56

virgin it: ii? FEEL. urea AUS: EFL Process . xks_meta_ingester streams metadata over socket. This process improves database performance. Instead of each xscore_proc writing to the database independently, they stream their metadata over socket to the meta_ingester, which combines it by plugin and writes to the database. Reduces the number of connections to and gives better control over table size. TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 5?

'?i?liirjgi??' .liti" all a Eli are USA, ER: 5 . register_metadata_tables moves tables from processing database of XKEYSCORE system to query database. . Works against the uber_index table up base_table_name, join_table 0 Base table contains common information amongt tables table) Extension table extends the base table . Registration process takes place in two phases: Register all base tables Register all extension tables that have had its base table registered TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 58

. 25a an .iti Eye; to) [Em ER, NZL ther Processes 1, . signal_acquisition_loopback process that feeds modified packets back into the system. 0 Front-end for packet recursion or any other process that feeds modified packets back into the system Reinjects back to front?end xfip Process is completel independent TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 5?3

it: is] ?ll IEEIL. m) [um 53% BAN, ER, NZL ,th?erProcess .. 3 - mpmr_server this is the map-reduce server for microplugins, which runs the ?Reducer? portion of GENESIS v5 microplugins. . Runs outside the normal processing flow, and will not affect the rest of the system. I It has a telnet port (5850) just like an xscore_proc. EECRET Si REL TU USA, AUS, CAN, GER, NZL 60

. at an .iti Eta; [Em ER, NZL ther Processes 1, - correlation_server_0 in-memory map?reduce server for correlation engine. . Each machine has one correlation_server, and every process_data_parent connects to every correlation_server xscore_proc 8GB by default 0 uses port 4321 TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 61

ma us; nus, can, Gee, NZL Processes - xks_comms_server a more efficient way to communicate with hosts within and outside an XKS cluster (not currently implement) . Automatically handles configuration for talking between slaves, master and overlord at site Configuration is needed to connect to the ?peer? on the path towards, other sites 0 Comms configuration lives in 0 Supports a ?quality of service? which ?fairly? distributes available bandwidth to the services that are using comms EECRET Si REL To USA, AUS, CAN, GER, NZL 62

a] a mat. mm AUS: ?3 BE .rih?erProcesses i 1 . xks_comms_server . Allow and Peer rules have a ?network? parameter which the comms systems uses to determine an ?inside? and an ?outside? in proxies. Comms system will only accept connections from address ranges it has been specifically configured to allow. 0 Every between 2 comms servers connection should have: ?bandwidth_rule? on each side, name doesn?t matter but both rules should usually have same bandwidth cap ?allow? rule on one side with a reciprocal ?peer? rules on the other side TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 63

as {??zll Ill @5121; ?g [gm Lima, @531}; ER, NZ 'th'er Processes . xks_comms_server . Example: If we have a site named connecting to xks?central over a 1Mbps link, config would be: bandwidth[world] peer[00] port=2412, bandwidth=world, network=external And xks?oentral would have: bandwidth[usl23] 1Mlops allow[00] bandwidth=us123, network=internal TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

a all it? to) meat, AUS, BAH, ER, NZL fI Pro cesses . Other process which is run from the Application Launcher. . GUld rescans content against fingerprints when a user clicks to View the content of a session. . tomcatsh web server used to host XKS GUI . sotftod124server downloads sessions 0 Gets called from the process 0 Works with any downloaded traffic that is SOTF TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 65

it at] liters TIE) [@339 aye, ER. NZL .?Sta?tistic Processes hm,? I . Other process which is run from the Application Launcher. - xks_server_stats sends to xks_system_monitor on Master and generates stats about the server itself. . CPU usage, memory usage, disk space, disk network traffic, etc. 0 Stats are fed to xks_system_monitor and the system monitor does magic with them. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 66

:S-Ell It? IEEJEIL. Tr@ 493% @9311, FL NZL Statistic Processes . xks_system_monitor collects stats messages from all over the system (front?end and back-end and the server itself) and summarizes them for forwarding. Optionally it can database stats locally. TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 6?

(if; Fl? fill Jill li??lL. Trle 5m @9531: II I Cronjobs XKEYSCORE uses a number of cron jobs to perform tasks. age_off_new.php Ages off metadata and content when the disk is near capacity, or when thresholds have been met. update_dictionaries Pulls updates from various sources. push_config Copies the lopt/xkeyscore/config directory to the slaves. rwc_post_to_pub.py Once an hour kicks off an update request TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 68

its: (ti: Fifi?i ran .i'ii' IEEJEIL. Trt?) @533], @331}, i I CRNTAB I . Crontab is the program used to install, uninstall or list the tables used to drive the cron daemon. . The crontab consists of age_off_new.php update_diotionaries push_config nNo_post_to_pub.py TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 6?3

as as:ataxia? at all it? IEEIL. W) was ?11m}, 6AM, EFL NZ TAB 3 . age_off_new.php 0 Options: -debug extra debug statements in the output 1* ?info extra info statements in the output -task_db explicitly state that the machine is a task host -web_db explicitly state that the machine is a web host ?nosleep use if you want to run now 0 This process ages off tables and archived data based on the settings in the xks.config file and the percentage of disk space used. EECRET rr Si REL To USA, AUS, CAN, GER, NZL 70

52%] EEL ?31339 BAN, ER, NZL TAB 3 . update_dictionaries This process pulls the necessarin files from various sources to update the dictionary. 0 Configure #[dictionaries] dictionary[0] type=royale, pd Idev/null dictionary[1] type=cadence EECRET Si REL To USA, AUS, CAN, GER, NZL 71

- it an re- use ear-33 em @3311, Mb CRNTAB .13 fl . push_config Transfers Master Configurations to its slaves. Excludes dot files, loadserver/packages", . force: option to to force push_config when not on the master TUF EECRET rr 3 REL TU USA, AUS, CAN, GER, NZL 72

til" en a mat. mom AUSrwc_post_to_pub.py The automatic starProc process is as follows: Hour 1: master asks whoever (say xks-control) for an update, gets the rpm, installs it, there is much rejoicing. The slaves asks the master for the at the same time the master asks xks-control, but obviously the master doesn?t have it, so nothing happens. Hour 2: everyone asks for an update again, this time the master has the rpm, the slaves download it and install and there is much rejoicing. The is installed and process_data_parent?s are restarted as soon as the is downloaded on a given machine. EECRET Si REL To USA, AUS, CAN, GER, NZL 73

TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

. .- r' -- $12121; 25319359 @393 @339 DeepDive What is a DeepDive? Why DeepDive? What does a DeepDive look like? Front?End Processes XFIP Promoter TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 75

as; egress? a e] a Eli. tom AUS: ?3 BE ?poive XKEYS c; - I XKEYSCORE packet processing solution 0 software handles all packet processing 0 No upfront filtering prior to XKEYSCORE . XKEYSCORE ?promoter? tries to promote richest/most interesting traffic All Strong Selectors Full take ASDF (User Activity metadata) Subset of GENESIS signatures . List managed by XKEYSCORE team in concert with collection managers and site engineers . 20% - 30% of site traffic is fully processed and can be found via XKEYSCORE search Typically does not include unknown or uninteresting protocols SECRET Si REL To USA, AUS, CAN, GER, NZL 76

vise it" all a mag vs) USA, ER: ?h Tf.l 5 Access to most relevant DNI data supporting SigDev and collection missions. Enables new mission capabilities Correlation) Session promotion can be and managed based on Genesis signatures, traditional tasking selectors and available resources Provides better scaling Drop unwanted data. Keep the rest and make decisions later and more accurately Better control of the processing space Instantiate new mission capabilities and dataflows quickly Troubleshooting and monitoring made easier Need access to ?raw? packets to support new mission Cyber, Bulk 0 Sessions can be displayed as Packet Bundles like Wireshark Si REL To USA, AUS, CAN, GER, MEL 7?

DeepDive . Whatdoes a DEEPDIVE look like? 0 XKEYSCORE full-take session processor (Back End) 0 High speed packet ingest: an end?to?end solution . Intelligent filtering to vary the proportion of traffic retained DEEPDIVE ?a i Front End Back End Packet Splatter Promoter Defreg I I a. A. Packets Partiai Sessions Fuii Sessions I EECRET ii 3 ii REL To USA, AUS, CAN, GER, NZL 78

Packet Splatter Ingests packets (frem files, frem the frem a capture card)- in a ma riety ef fermats. :cFip Feet reassembly cIf streams*, and and UDPJIPUE: streamsf?. Reassembly ef streams frem less cemmen stacks. Premeter Rule-based filtering cIf reassembled sessicns, based c~n keywerd, C?untry cede er appidg?fingerprint. Defrag Fullyr rebuilds sessiensff a: earr- gal eat. Tr@ [Em @5311: 5 FL XKEYSCORE Front-End What it's called What it dues What it means If it's a packet stream, it can prbbably be fed inte a DEEPDIVE. DEEPDIVE sessienizes everything befcre making a keep?drep decisien. DEEPDIVE intelligently cheeses the mast useful traffic fer retenticn. Eneugh centent available be de full descent at the Back End EECRET Si REL TU USA, AUS, CAN, GER, NZL 7?3

at] fill lasts m) [Em awe, BAN, ER, NZL I - . Packet bundles Preserves original packets and packet order 0 Preserves information that is lost during sessionization . Original pcap available in the XKS Viewer . Packet API 0 Microplugins can iterate over raw packets Mioroplugins can use information that is lost during sessionization E.g. timestamps, flags, checksums . Packet fingerprints Fired based on observations xFip has made E.g. large sequence gaps, TTL variation rr Si REL TU USA, AUS, CAN, GER, NZL 30

a a] a REL its) use, AUE, ER: I Filters sessions prior to back end processing - keywords, regex, country code, appids* - SIGDEV: promotion rather than strong selection . Set the focus of the back end - traffic types of interest allow appid chat.* - regigns of interest allow country_:ode PH I legal/policy constraints him ?3?113 . Set the width of the access aperture . promote 20% of 20 signals? . promote 100% of4 signals? - Set the length of data retention .- promote 20% and keep for 3 days? promote 30% and keep for 2 days? EECRET Si REL To USA, AUS, CAN, GER, NZL 31

TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 32

E1931: ?i?ie??it'l. @1333 Script I Usage Voptions General Commands Services Actions Options TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 33

it: a] .iti' liters. vs) [Em ave, BAN, ER, NZ .XKS'ptions - . Usage: [options] <command> 0 Try 'xks help <name>' to get help on a specific service or action 0 General commands: in? services list available services actions - list available actions Ir dependencies [invert] - shovv service dependencies Ir help [items] - print help on services or actions 0 Services (specify one or more service names or I start {services} - start the specified services Ir stop sservices} - stop the specified services r? restart sservices} restart the specified services status sservicese - print the status of the specified services Ir setup {services} - setup/configure/fix the current install TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 34

a? Eli IEEJL. W3 [Em 624.13%, GER, NZL . rail 0 Actions: Ir accounts_report - sends an email containing accounts usage to the specified users In? add_admin sets up a local Linux user to administer XKS change_db_passvvord - changes the XKS database user?s password and updates all references to it cluster - cluster actions compile_genesis - compiles GENESIS signatures disk_check - get raid and disk status ext4_format format partition and convert to ext4 filesystem ext4_upgrade - convert to ext4 filesystem vvhile preserving contents of (no formatting) fetch - fetch a remote file force_register - force metadata table registration info shovv cluster information instal _slave install a slave machine in this cluster local_tagging checks and/or loads tagging file 35 TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

1'7 ,xks? Actions . Actions: monitor mpmr_register onall powertower proc query query_dispatch rac reload_dictionaries search_fields show_config switch sync_accounts tail tasking_dump TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL ?L?Iili12541r?'i' it: or] Eli TIE) MBA, 495% .-- - 1iriew XKS monitoring messages via activemq force table registration - run a script - run a command on all machines in this cluster - configure or run a powertower command - control XKS processes on this cluster - display query status or submit a query command line interface to the XKS - access remote admin ports - force running processes to reload dictionaries push configs or files to slaves populates user settings with search fields - show values from xksconfig for specified keys - query or rebalance data switch user accounts (except for classifications) - View realtime logs - print out the contents of the kaasking and xksTasking_yoip databases. can, NZL t; 86

Actions top - display system performance update_dictionaries update all XKS dictionaries Ir update_gui_help update the 'help' pull downs in GUI users - display the users currently logged into the GUI In? version - showr XKS version information watchdog check and (re) start essential XKS processes. workflow - manually submit a workflow Si REL To USA, AUS, CAN, GER, MEL 3?

if; ?rIZi-Zigz? {st-tug, (Ell: rm ptions 0 Options: -verbese print extra information to the screen -debug used for debugging script problems TUF Si REL TU USA, AUS, CAN, GER, NZL 38

General Comma vigil 'E'afElEiif??lElEi?J' it: FEEL. so seas AUS: EFL . Type: st help services This will list all available services: first initialization service that runs before all others virus_scanner sets up virus scanner, assuming tarballs are present. enables on the master if mailorder is enabled distcc sets up distributed compiler service slash_proc setup optimal Iproc parameters myricom handles installation and configuration 10GigE network cards home sets up the home directory for the user account check there is a working compiler on the system upgrade updates configuration files when upgrading to a new version of bashrc sets up bash environment variables beacon sets up monitoring beacon based on xks.config tt checks connectivity to TRAFFICTHIEF server TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 8?3

vitfgl it: of] it? lie-EL so) see, AUS, BAH, ER, NZL ,st? General Comma 0 Type: help services This will list all available services: . sendmail configures sendmail for use with . role_files this service installs role-specific files . issue sets up the mandatory login warnings - royale_with_cheese setups automatic updates . configure based on xks.config . ink_summary sets up link summary GUI . sets up xks-specific mounts . server_certs sets up server certificates for SSL applications . openoffice installs and configures OpenOffice for use in the st GUI - init_d sets up the init_d services . resolver sets up resolver config . sets up PHP related stuff. Except php.ini . sets up xks-specific configuration EECRET Si REL TU USA, AUS, CAN, GER, NZL ?30

vigil 'E'LfilEiif?'?E?Ei?f it: ll? REL. so urea AUS: EFL General Comma Type: help services This will list all available services: . sets up GUI configuration files . voip sets up voip processing . crond ensures can use cron and sets up cron jobs . configures the secure shell service for use with . license checks for a valid license file and if one isn?t found prints a message . syslog configures the syslog service for use with . all processes log to lvar/Iog/xkslog . dictionaries checks status of any configured dictionaries . cluster_check checks network connectivity across the cluster - autofs start, stop, restart automounts - loadserver start, stop, and setup loadserver - directories sets up directories used for - auditcl no help available EECRET Si REL To USA, AUS, CAN, GER, NZL ?31

visa etisijzaicia?v a to] it? IEEL. as sea, AUS, BAH, ER, NZL .st? General Comma a Type: help services in? This will list all available services: - ldap no help available . sets up the server for use with - disks checks status of disk partition used by - databases maintains database scheme consistency - local_tasking reapplies local tasking if necessary . workflovvs sets up default workflovvs - category_throttle overrides default category throttle settings based on overrides specified in xksconfig . enrichment_tomcat sets up enrichment tomcatjava application server . plugin_setup populate plugin database tables from files, appy default plugin config specified in xksconfig, apply overrides from xks.config, regenerate plugin config files from database - no help available . tomcat sets up tomcat java application server - sets up service EECRET rr Si REL To USA, AUS, can, GER, NZL ?32

General Comma val assess lit" a] a was verses AUS: ?3 BE . Type: help services This will list all available services: file_input sets up directories and database entries needed for file- basedinput age_off_db the database (xs_task_db.age_off) with xks.config?s settings for content and metadata. The values in the database will be unconditionally overwritten with those found in xks.config db_connectivity verifies connectivity to critical databases sets up language packs ul_age_off? sets the maximum data retention time to a little over an hour in UL mode. sets up for use with SOTF input app_launcher? controls the app launcher, which is responsible for monitoring processes and them as commanded from the GUI processes_setup configures processes based on specifications in xksconfig comms sets up the XKS communications system configuration adunnr?n??l? 93

f. r' -- 32121 @133, I General Commands 0 Type: help services This will list all available services: . enclace handles all the installation and configuration for Endace Dag packet capture cards I last cleanup service that runs after all others TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

aat It'l?i' Iiuli??ie?a attuiErE-t, 333% ?xks - Services 0 start start . stop stop I: restart restart 0 status status autofs setup plugins TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL ?35

1001 00! H'h?'hr?l1lfi-liv - Irl . . 1 HUI. H. w? 'j irl, . U. 3 I onall ?ps ?ef grep xscore grep ?v grep? force_register ~r immsm?g? TD CAN, GER, NZL - r13: push_config -force Usage: <0ptions> <src> <dest> update_dictionaries Usage: update_dictionaries versio__ info run]$ 1.5.9?55 TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL [aper?tlxkavr?l run]$ farce_register 1'5. E1 41:} ?36

IOI mm ?mF, om mo. Harm" I. TGF SECEET SI REL TD LISA, AUE, BAN, GER, NZL . um: ?.001 mm ?Im too I, -. Q. nox' I W- c? ?olounmomm 31? u. "No..2. 110! - In' query sewers [eper@tlxkevr01 run]$ Eke query eervere t13kevr02:q0 2a 1&3 90m 54w 2012?12?05 15:0? tl?kevr03:q0 23051 Us 2012?12?05 2303 DE 2012?12?05 tlervr05:q0 225a 03 2012?12?05 tl?kevr0T:q0 230a De 2012?12?05 tlEkSerE:qD 225a 05 2012?12?05 tl?kevr09:q0 De. 17'31?1 2012?12?05 t13kevr10:q0 220a De 2012?12?05 230a Us 2012?12?05 tl?ksvr12:q0 2253 05 4D 0w 2012?12?05 e=eweiting dispatch, e=eent, 11=11emi5r w=werking timeetemp ehewe earliest Submitted but unfinished query current time: 2012?12?05 10:02:09 TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

- .. [ma ammo: hwmuwj- moi um um: i' ?9.10.101 IOIJIO 3; wk E: ID 0' 'Unoum mmom IIUIQHIDI 0' mm: . proc IIO In' I D. 1 [CpCrUtlEkEerl run]$ GUI query_prCC Cli regiEtCr_mCtEdEtE_tEblCE CEUU Eignal_ECquiEitiCn_bEEC EadE?CE_taSki?g_prC Cur Ed Eth_diEt file file_input_prCC mp mailCrdEr_prCC tCmCEt.Eh ECE pd# Emi EkE_metE_iugEEtEE EkE_EyEtEm_anitCr qd query_diEpEtCh Run full? tC Ewa full listing I tlEkEerl GUI Cli CME Cur file mp qd qp tlEkEerE CEUU qp tlEkEerU CEUU qp tlEkEerU CEUU qp tlEkEerE CEUU qp tlEkEvrBE CEBB qp tlEkEvrBT CEBB qp tlEkEvrBU CEBB qp tlEkEvrBU CEBU qp tlEkEvrlB CEBU qp tlEkEvrll CEBU qp tlHkEvr12 CEUU qp tlEkEvrl? CEUU qp CEUU ?unll" Fl'i rr- rl'l fl'l fl'l fl'l II Li- TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

)l?l mm: 9' 9 01 town'51 Itaeg4-: ?hs- a- . proc full 9 run]$ PIQC full id 14 723 654 app launcher status: RUM hastname tlxkavr?l tl?kserZ [pid EDTDEJ pragram cadence_taaking_prac enrichment?tnmcat.3h file_input_pr?c mail?rder_pr?c query_di3patch query_pr?c register_metadata_tables Signal_acquisition_baae strong_?elect?r_targetiug t?mcat.sh Hkg_camm3_server Hk5_meta_ingester Hkg_server_stat3 Eks_3ystem_monitor carrelatian_server_? mpmr_server pracess_data_parent query_prac regi?ter_metadata_pables signal_acquisitien_baae E?tf_di3t Hkg_c?mmg_3erver Eks_meta_ingeater Hks_server_stats TGIF SECRET SI REL TD LISA, AUE, CAN, GER, NZL argumentg ??myfdi ETD --pddg IE . . ??l?gleuel errmr -f gene . . ??l@glevel debug ??maH?mem ED --lmglevel errmr ?f gene. e. . . Signal_acquisiti?n_lm0pback -f -i 100.. TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL cummanded FUN JUN JUN JUN JUN JUN actual

In 1 1m] :3 mi" am?? um um ?nga?. HEEL Te CAN, GER, NZL mm om loo: iolslo . ..- 0m IDIOIO at 1' . query In' a 1 [eper6tl?kevr61 run]6 er query id ueer type search Start search step duratien statue 66256261 http_pereer 66:66 1222212 66:66 1226212 66256163 full_leg 66:66 1223212 66:66 1226212 66256155 full_leg 66:66 1224212 66:66 1226212 66256126 gee_1nfe 66:66 11236212 66:66 1226212 66256652 66:66 11221212 66:66 1226212 31:23:36 66246663 full_1eg 22:66 1223212 21:56 1224212 66246666 full_leg 66:66 1222212 66:66 12262 32:15:; 66244233 eetegery 66:66 1125212 66:66 12262"6 73"7'? 66244135 full_1eg 66:66 11226212 66:66 12262"? 2::llif: 66244666 http_pereer 66:66 1125212 66:66 12262"? 66243666 http_pereer 66:66 1125212 66:66 12262"6 . 66243655 deeument_mete?ete 66:66 11221212 66:66 12262"! 66243665 eerreletien 66:66 1125212 66:66 12262"? 66243463 eerreletien 66:66 11221212 66:66 12262"* 66243661 66:66 1121212 66:66 1226254 66242663 ueer_eetivity_exif 66:66 11221212 66:66 12262"4 66242413 http_pereer 66:66 11226212 66:66 12262"? 66242315 full_1eg 66:66 1224212 66:66 12254"- There are 16 queries in TUF EECRET 22 3 6" REL TU USA, AUS, CAN, GER, NZL

In I 10' It'll ll!" 5 ?lmF' m. um. 3 REL TD USA, AUE, CAN, GER, NZL not Ion-'In' i - gram m; v- Mammal 0"?101 .. DDIIDIOIO "3 '00? . no I 1 . query detail [eparutlukaur?l ruu]E uka query de:ai1 ?1 I. Query Summary Uaerid: Tyye: Searching freu te 1255::2 Duratieu: Prierity: 5 Cancel: Hie] Haa Reaulta: Haa Time: EDJD Query Name: amater3_e Eyl?f?ly?EA Where: datetime a: 'E?l?-ll-EE AH: datetiue i: DE DE AND email RED dumaiu Thutuail.n:em' Query SZatua heat databaae atatua tlakarr?l q? f;uiahau tlakarr?E q? augaiuy tlakarr?? qD f;uiahau tlakarr?e q? f;uiahau tlakarr?? q? f;uiahad tlakarr?? q? f;uiahad tlakarr?? q? f;uiahad tlakarr?? q? f;uiahad tlakarr?? q? f;uiahad tlakarrll q? f;uiahad tlakarrlE q? f;uiahad tlakarrl? q? f;uiahad tlakarrle q? f;uiahad q? f;uiahad tlakarrl? TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

lirali'IL TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 102

smug? Lesson bjectii/e? Executables mesqls Vonall kas onall kas monitor Vsotf_stat kas top Web Status Additional Monitoring EECRET Si REL To USA, AUS, CAN, GER, NZL 103

1- ?e @339 mu i Executables . System monitoring can be performed from the command line using the following executable commands: onall onall sotf_stat top TUF EECRET Si REL TD USA, AUS, CAN, GER, NZL 104

ma us; nus. can. GERThe bash shell script can be used to execute statements from in the /0p t/xke yscore/b in. sh ells/sysadmin/mys directory. The most commonly used options in are: . status displays file-based input statistics. 0 speed displays the total file based input processing rate (Mbps) . Speed1 displays file-based input processing rate (Mbps) per input source. 0 speed2 displays file-based input processing rate (Mbps) per xkeyscore processing server. 0 count displays the count of input files in the new. working, error, and done states. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 105

H.101 1 535:,sz Jnrul mm Ham ?Vi0:01status [aper?tlxkavr?l run]5 Etatus statua ?aunt?i? sum[fileaize] primrity bitrate Mbpa HULL TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 106

at man no: mm me001', :00: mm [L?f'oio? :01 It a a an lira or I J9 a ?hl?T?F SECRET 3 REL TD LISA, AUS, BAN, GER, NZL a +n-L ?4 Holman-u: . [?ail 0.10'010 '01 Im?l m. . m' I . onall ?xks status? Eks small *sks statusT ?Hks status? :ta??i status E2. status status _n status 1.- . . - I.-. v: status - 1.- PF. I -.I- -- TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

ID . 1 -Llf I I 1w: 1 cm .00. Wm? HEEL TD CAN, GER, . ,l Iml um um ?'i?flnule? 1m (1 '0 Icol .?h'?mh?ni 1L3 ?1"3 'i91 . ?li?la ohsuA-Il I 10! T) If I This script will monitor your front-end processes. . Type: mentor or monitor to receive the help menu Gummand IWame I?escripti?n. Ennfig Cenfigure this utility ?ataflem;all FrentEn? Datafl?w Menu {Len t; .1 c: IE: ac}: 91rd [lat 31f. 1 mg: i-lre nu :men n. ?vii em til] i me :n n. Packe Etc-{g 111:2: i ti In rut; En Precess Data [Back End] ?uitIEHit Server Etats lininput; [Back Emil] HHS tea)" Hfip [Brent End] TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 108

LISA, AUE, BAN, GER, NZL mm mm .t'j'ol??noHolman-Type: montor to receive xfip stats 5e; Eva-:- r: View Free-:5: dl'aszen-nt a1: i+:rn Rat?:- Luna-:lingk Fee-ute- arc-urn: Punt tr. Fragments?ts. D.DD CLUB El- IJIJ ELEIIJ El ELDD TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL

aria a: at] it? IEETL. Tire [@3339 age, ER, NZL .sothtat . The sotf_stat command is used to display the SOTF (streaming object transfer format) input statistics for an entire cluster. I The statistics include total number of process_data?s running on the cluster, session input rate (sessions/sec), total bytes input (Mbps), and total bytes output to process_data(s) (Mbps). TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 1 10

it} astr- gal lit @3121; Two [Em we, @5311: 5 FL sotf_stat . To execute the sotf_stat script: . Log on to the server and Open a terminal window. . Type sotf_ stat because the command is in the path 0 Type 8 to toggle the summary statistics View from total statistics to individual host statistics. . Type to quit the program EECRET Si REL To USA, AUS, CAN, GER, MEL 1 1

I a u. v1- m] HM.qu TQSECHFT DESI REL TD USA, AUS, BAN, GER, NZL J. I mm Holn?m? m- I- . in m- --.- a. unity,? .1 0' 11 l?DILIJDI m: cm (1-113: 5? .. :uww_ .10 '11" 1 .., - if The sotf_stat script lists the hostname, number of process_data?s currently running, Mbps, number of sessions, and number of tes. 5 IQ. ?ees in ?i?l?lBEE - Heetneme I .- I I. Fl {31. :7 Ln 1-: en. La! r31Eesslens: TUF EECRET 3 REL TU USA, AUS, CAN, GER, MEL 1 12

ma: TD . "?u-3T . 1 . It"01EX.I ?3 ?ii . . Iul IUHJ lu .J :lii'w at 1? . .. u, 11*" war - -w i The top script lists the hostname, sotf rate, number of process_data?s running, the of CPU, and of IO wait. mhe?eevg?g TUF EECRET 3 REL TU USA, AUS, CAN, GER, MEL 1 l3

moot nan (?lmm I n1 10:01:: 39 c4 '10! - n- [Sper?mhakIOI MIMI doll I 2' II Lu" 5 TE: [aatamaqi 5 la:f? {autamatie. Tl Ex'r I 5 1a: {autema 5 13:3}: {autema 13:? -434 u] mi I rq__n _1 II -: 15:1 u: {It 11:: {a {a II II Ii-l i r- .i :2 Ili- 't IL h; .13 l?I- ll {aatpmatip?j a 1e:aa an {autpmatie?j HyuI i . 0' ?it'r?T {al?n?gimlt ??ounmomnno tail erE regiater metre. a; F: a rapal? rgpair mha:55Vr.. IEpair falLE mhakaavr?? regiater_yn rapair failed f??aecam 3 REL Ta LISA, AUS, can, GER, NZL 't mhakaavr?? regieter_metadata_taplea lids: repair failed mharaavr?? repair failed mhakeevr?? E5 mhakaevr?? regieter=metad repair failed mharaavr?? regieter=m?r repair failed mhakeavr?? regiater meted? repair failed TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL raetf diet ta 1: areaiater 114

TUF EECRET 3 REL TU USA, AUS, CAN, GER, MEL 1 15

2:54: egg Trial @533, @3339 GEL - Lesson bjectii/es Common Troubleshooting techniques VFull Disk VSotf Problems VProoessing Problems VOutputs VQuery Problems VDireotory Permissions TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 1 16

atria .liti" @il a Eli at) USA, ER: . . .. 4 Fun-(xks tail) Relevant error messages can be viewed in this file. This directory may fill the disk, some known reasons are: . process_data has lost its connection with the sotf_dist and is continuously trying to reconnect to sotf_dist. error may have occurred and a detailed message can be found in the file /var/log/messages. . Corrupt tables in the insert database. 0 Check to make sure the age_off_new.php cronjob aged off old metadata and content. TUF EECRET Si REL TU USA, AUS, CAN, GER, MEL 1 l7

?rs it: an] ill legit. so [Em em, BAN, ER, NZL I Disk Full continued. .. - - . lexportldatalxkeyscorelinputs If there are too many files in the directory: file_input_proc may be running improperly or not at all. Verify that file_input_proc is running from the command Hnetype: . ps ?ef grep file_ grep ?v grep . proc The file_input_proc may need to be restarted. . No new files in the directory: 0 The directory may not be cross-mounted properly, if automounting is used. TUF EECRET Si REL To USA, AUS, CAN, GER, MEL 1 18

a er] a REL its) USA. AUEare filling and (30 and/or q1 maintains its size. register_metadata_tables may not be working properly. Restart process and watch the databases to see if it is transferring files or run the process by hand to troubleshoot further. a If or (H is filling, the age_off_new.php script may be running improperly or notataH. up First run the command: ps ?ef grep age_ I If script isn?t running, try running it by hand. I If script is running, then stop script and try running it by hand to see if there are any errors. EECRET Si REL To USA, AUS, CAN, GER, MEL 1 1?3

was as NSF, AUS: BAN: GER: -.4 Problems . Can an sotf_input_proc run with a file_based file_input_proc? . Yes. Both input types can run on XKEYSCORE given that each are independently configured correctly. . Can file-based input be disabled so that only sotf_input is processed? . If moving from file-based input to sotf_input, and no additional file-based input is expected, the plug-in for file- based input, db_input_file_handler, should be disabled. a From the TERMINAL WINDOW: In? Stop all the processes stop all Change to set file_input to II- Setup the config setup plugins, setup processes II- change to slaves push_config II- Restart process_data?s proc restart EECRET Si REL To USA, AUS, CAN, GER, NZL 120

vigil it: ll? REL. no set, AUS: EFL Problems contin a a . ls XKEYSCORE receiving input? To verify whether XKEYSCORE is receiving input, run the sotf_stat command to get the current input statistics. 0 If no connection is visible, from the command line: 1. Type telnet: looalhost 5042 . Output statistics for the specified sotf_dist If running, type ps -ef grep sotf_dist Determine if sotf_dist?s are listening on the specified port: Type telnet localhost 5040 If command is refused, the sotf_dist is not listening on the port. Continue with step 5. 5. Type netstat ?a grep 5040 If a connection is established for this port then most likely the sotf_dist is listening on this port. spam EECRET SI ii REL To USA, AUS, CAN, GER, NZL 121

as eats:artists? it: all IESEIL. Wu) [Em ?11m}, 6AM, EFL NZ ST Problems continug . netstat will tell 0 sotf_dist is listening for connections 0 If connections have been made to the sotf_dist If we are ?backing up?- if sotf_dist is running but has no process_data?s connected to it, it won?t be able to send data anywhere, so eventually its network receive queue will get large. Ideally, the receive queue should always be 0. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 122

as? an ill @121; mag @2311, a, NZL STF Problems continue a . Is the process_data_parent running? 0 At least one process_data must be running and with the sotf_dist for it to receive input If problems continue, run the sotf_dist in a terminal to further troubleshoot and identify error messages. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 123

a FEEL. meat, AUSerrors or too many! errors display when performing the command status?: 1. First try, Cleanup, in a terminal window. 2. Type status 3. Type x5 task_db; to log into database and use the xs_task_db database. 4. Execute the following command: delete from tar_files where status=?error?; 5. Exit out of the database 6. Type mysq' 3 status There sheaidvneelenaer error files. 124

aim it: FEEL. urea AUS: EFL ,Prooessing Problemsfl'_ . The heart of the XKEYSCORE processing engine is the xsoore_proo with related plugins. . Input to the xsoore_proc is either file?based and from an fi e_input_proc, or streaming from an sotf_input_proc. . After processing, the written metadata to the insert databases can be sent to a follow on system for additional processing. EECRET Si REL TU USA, AUS, CAN, GER, NZL 125

iasjE'? :S-Eil IEEJEIL. Tr@ 493% @9311, FL NZL . p. cessi r0 00an . I How many process_data?s should be running on a host? 0 From the XKEYSCORE GUI: Click ADMIN Processing Computer Resources Determine how many process_data?s are configured to be running on the specified host. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 126

13:33:91 on: .00. . .t W- m" 1 ECHEIH s: REL TD LISA, AUS, CAN, GER, NZL M: x} K- . 1m' .(ni I. sl- ?Hart{fix?xix43:min): 141:.- :1 a: . 1C: . i: 131' f? . How many xscore_proc?s are actually running on a host? 0 Log onto the XKEYSCORE server and open a terminal window. 0 Type p5 ?ef grep xecere grep ?v managed_ l3333i313-33333]: p3 -Et' 3333 333333 3333 3333:3333._ 3333 115355 15 333:3 2' [7:35:24 3333333133": "3:13:33. 535:2 -- 3. "3231135313 31335 "3311:1313 2551.335 255E113 3333 19:33: 3' "13331333133333 [0:05:31 333333 2:11.13 -?333t. "3:333:51 appirl -- "333133 13331 -El 353: [3333 -- 2: "33313.31 3:333 "3333 33pm -- "333133 13331 -El 35:1: [33:3 -- EI 3:333 333333 3333 "3333 5332i "33333313133331:] 3:]33113 -- --reni33 13331 -E: til: 333:3 -- EZ 33333 TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 127

'E'ifElEif??EiEi?I' it: FEEL. sets AUS: EFL ,PiO'Cessi ro sir/?mime: . xks_app_launcher is running, but not starting processes specified in the Computer Resources window? 0 This may indicate that the xks_app_launcher is defunct. Use the kill command to kill the app_launcher and its related sub-processes: >Type pkill ?f app_ . lfa PID is not being specified, use the pkill command. The ?f option kills all of the sub-processes. Type ps to look for the new xks_app_launcher process. TUF EECRET 3 ii REL TU USA, AUS, CAN, GER, NZL 128

rt: a] ll? Eli. Tito (we. ate, ER. NZL .Pfoce ssi ro 3 tin If, after performing the procedures, the st_app_launcher is still not starting applications: In a terminal window, manually run the problem process to see if there are any error messages. The xks_app_launcher on any host is dependent on the access of the xs_task_db.proc_resources database table on the master. Verify that the specified host can access the master?s database and /opt directory. On the slave system type xs_task_db ?h {masterhostname} . performs a remote server login EECRET Si REL To USA, AUS, CAN, GER, NZL 129

[as til at] .iti' IESEIL. Wu) [Em ?11m}, 6AM, EFL NZ PrOCessing Problems find/e. . To test the xscore_proc, type: telnet <prooess host> <port number> Optional commands to assist trouble shooting are: 0 prints the processing rate for the single xscore-proc. . sh displays dictionary hit statistics. 53 displays statistics on the internal plug-in processing rates. 0 help there are many commands and can be described in the help menu. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 130

a a] at? REL tie} USA. AUE, ER: .Processi ro ?pntin . If the process_data_parent continues to deny access through the command port, and input still has not started processing, check the input source. . Run the process in a terminal window with the argument ?-loglevel debug, to view debug messages. . The command port also provides processing rates and statistics for troubleshooting performance issues, outages, and general administration issues. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 13 1

vigil 'E'afElEtf??lEiETf it: it? FEEL. urea AUS: EFL ;futputs - Mai/order . . If there are no new files in the MAILORDER directory, MAILORDER may not be working properly. Possible causes are that: 0 Files are being written to the wrong directory or it is not configured properly 0 Permissions on the MAILORDER directory will not allow MAILORDER to move files TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 132

a 5' Elfil?if??l?l?'?' lit? REL"ll, . I I Di Spatoh . Query dispatch is the process that submits search jobs to search databases and propagates the status of the search and the results of the search back to the web server. - After submitting a new query, Search Status window displays a summary listing query name, date and time submitted, number of databases complete, and number of results. TUF EECRET 3 REL TU USA, AUS, CAN, GER, NZL 133

932le ill Esta; no use, 52MB, @1111, NZ -ju'ery Problems 1 The query never moves to the finished state. 0 If a database outage or a oomms outage occurs, results will not be reported from the single system. However, results from all other databases will return properly with the query results, but they will not appear in this state. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 134

is eats:grizziai?? i515" Eu] IESEIL. Wu) [Em ?11m}, 6AM, EFL NZ ,;uery Problems 3 . Query job status is stuck in awaiting_disbatch. . If a status appears stuck in this state, the query_dispatoh may not be running on the web server. To determine whether it is running: Type ps -ef grep query_ 0 If the process is not running, restart it from the XKEYSCORE GUI or troubleshoot the xks_app_ aunoher. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 135

?fti-EJE' ma NSF, AUS: BAN: GER: Problems _'77 0 Another cause of this scenario is that a query database may have hung up the query diSpatch process. Check the progress of queries on the query database hosts by viewing the table sdb_queg/_jobs in the query database, which tracks the status of queries: Type qt] Type select: status, count: frem sdlo_query_j obs where group by status; 0 The select statement displays the current state of the queries on the query host. If many more queries appear in the new state when compared to other query databases, begin troubleshooting the problem query_proc on the specified query database. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 136

attijztit?fa??' a tail ll? to meat, AUS, BAH, ER, NZL guery . Processing . The query is in the sent state, but never appears in new. 0 After the query_diSpatch process disbatches the query, the status is moved to sent. A query moves to the new state when the query has been placed in the query processing queue on the query_host. If a query does not move to the new state in a reasonable amount of time, the connectivity of the database should be tested. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 137

assess? it" s] a list mm AUS: tie?: ?3 BE _,uery Processmg i, 1 0 To check the progress of queries on the query database hosts, View the table sdb_queiy_jobs in the query database, which tracks the status of quenes: Type qO Type seleet status, count from sdio_query_j obs where group by status; . The select statement displays the current state of the queries on the query host. If many more queries appear in the new state when compared to other query databases, begin troubleshooting the problem query_proc on the specified query database. EECRET Si REL To USA, AUS, CAN, GER, NZL 138

a oi] as REL as man, AUEThe query appears in the new state, but never finishes. 0 query is in the new state, has been received by the query host and placed in a queue waiting to be processed. Queries can become backlogged with a large number of queries waiting in the new state, though the query_proc is processing the queries properly. It is hard to predict the time to work off a query backlog, but using the following select statement the status of queries for the current day can be checked for processing trends. TUF EECRET Si REL To USA, AUS, CAN, GER, NZL 139

r: en m) were, AUS, BAH, ER, NZL ,U.le guery . Processing . To display the number queries in each state for the current day: a Type eeleet statue, Submitted,(UNIX frem edb_query_jebe where (detetime_eubmittedb(new() INTERVAL greup by status; . To display the number of queries processed per hour for the current day: 0 Type select RS queriee_per_heur frem edb query jebe where and (deEetime_submittedb(new() Der)) es Beekleg greup by Eeneel; . If processing properly, queries can take hours, if not days, to complete based on the backlog and the processing trends. EECRET Si REL TU USA, AUS, CAN, GER, NZL 140

a @ii a Eli are LISA. ER: .Rfetriev i Metad ata . Queries complete but there are no results. I If queries complete, but no results are visible, verify that the date range of the query coincides with the collection date of the data. If using test data, test the query system by putting the start date range at a year or two older to assure it is not old test data. . Verify that query metadata is in the query database by checking the contents of the /expon?/da ta/xke yscore/m ysql/{q uery_ db directory. EECRET Si REL To USA, AUS, CAN, GER, NZL 141

ride @ii a ma; LISA. ER: .Rfetriev i Metad ata . Queries complete and metadata returns, but there is no content. I The metadata in the XKEYSCORE viewer displays the host and directory path of the content file. Verify the content file exists using the ls -I command. Trace a dataflow issue if the file does not exists. If the content file exists, confirm the daemon is started on all slave systems. To confirm the daemon: 1. Type su oper 2. Type status - 3. Ifthe daemon is not on, type start TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 142

aisle? it: a] liti' IEEJL. to use, we, BAN, ER, NZL ,uery Results i ?l t; . To troubleshoot problems with metadata or content from a query, it will be necessary to retrieve the actual content, since recreating the problem is very difficult. This can be accomplished from the XKEYSCORE GUI. Click RESULTS and begin a search of the questionable queries. TUF EECRET Si REL TU USA, AUS, CAN, GER, NZL 143

""311? TUF EECRET 3 REL TD USA, AUS, CAN, GER, NZL 144

XKS System Administration

Enter your email to keep reading for free.

We’re independent of corporate interests. Will you join us?

Enter your email to keep reading for free.

No ads. No corporate BS. Skip the propaganda and donate to keep The Intercept going strong:

No ads. No corporate BS. Skip the propaganda and donate to keep The Intercept going strong:

We’re independent of corporate interests. Will you join us?