RWC Updater Readme

SECREFHRELTD USA, AUS, CAN, GER, NZL Rayale with Cheese Updater Tn Guide Setup and cnnfiguratinn nfthe Rnyale with Cheese Updater takesjust a few steps. the steps belnw tn quickly "get up and running.? Refer tn the Read Me greater detail abnut each nptinn. nn as the user nper, and fn nw these steps tn configure the Rnyale with Cheese Updater: 1. (SHRED At the cnmmand line within any type and then press Enter. The xks.cnnfig file will npen. Next tn the nyc_publisher nptinn, type the IP address nf the Publish er which the current Subscriber (Master nr will receiye Nate: Alternatiyely, ynu may use a name instead nf an IP address if ynu are sure the current machine can resnlye the name nf its Publisher. Type :wa and then press Enter tn exit xxs.cnnfig. (SHRED At the cnmmand line, type xke eetnp rxe lxke eetnp rn in yersinn 1.5.3} and then press Enter. This sets up directnries and yariables that are used internally. (SHRED At the cnmmand line, type rxre and then press Enter tn get tn the the type exe frwn_pn et_tn_pn]n . and then press Enter tn run the rwc_pnst_tn_pub.py script. This sends the the Master nr tn its Publisher. nn as the user nper, and fn nw these steps tn custnmire PublisherfSubscriber cnnfiguratinn. A list nf PublisherfSubscriber nptinns can be fnund at the end nf this dncument. 1. 2. 3. At the cnmmand line within any type and then press Enter. The file will npen. a. Cnnfinn that 3 Publisher IP address nr has been entered beside the rwc_publisher nptinn. b. Next tn the rwc_mnx_chunx nptinn, type the maximum allnwable ?chunk? sire nf 3 Publisher file (in bytes} that is manageable by ynur Subscriber nnde. example, tn change the maximum chunk the default size tn 1MB, ynu can change tn rwc_mnx_cfiun #1000000. l'y'lnre details regarding rwc_ max_chunk are nn page E. Type :xrql and then press Enter tn exit At the cnmmand line, type efg and then press Enter tn get tn the cnnfig Classified By:- Deriyed Frnm: NSAICSSM 1-52 Dated: Declassify Dn: EDSEGEDZL USA, ALIS, CAN, GER, NEL

SECREFHRELTD USA, AUS, CAN, GER, NZL From the config directory, type vi aka .rwo . oonfig and then press Enter to open the file. Editing this file is optional. If no edits are made to this file, then the configuration automatically will get and publish eyery module that is ayailable. El. b. Change Publisher options within Customize any pub_}ot option to indicate whether the current machine will or will not publish a giyen package, to its Subscribers. For example, if pub_metodotojorworder=troe and you do not want the machine to publish any metadata forwarder packages, change the option to read: pub_metaciata_forwarcier fal ee Customize oddfremoye options: i. To add a file, type: ero="relfpatbftoffile", deet="relfpatbftofdest", where is the path to the source file including the file name}, dest is the destination directory of the file, and is the classification of the file being published. ii. To set a file to be remoyed from the Subscriber?s machine, type: rm_file relfpatbftoffile See pages and 3 for all Publisher options. 6. Change Subscriber options within El. Customize any option to indicate whether the current machine will or will not accept a giyen package, from its Publisher. For example, if sub_metodoto_forworder=true and you do not want the machine to subscribe to any metadata forwarder packages, change the option to read: eub_metaciata_forwarcier fal ae Customize sub_new_moo'uies to indicate whether a site can or cannot automatically receiye all new modules during a giyen update. The options are true or false: subscribe new modules=true subscribe new modulee=falae Use the notify_email option to enter the e?mail addressfes} of anybody who should receiye an e?mail message when new modules are ayailable but that haye not been automatically downloaded. Separate each address with a comma. For example, type: where addresses is an e?mail address. 2 sEcszsELTo use, wus, can, see, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL d. Set ignere eptiens te indicate te the Publisher that this specific Subscriber dees net want updates fer the specified filesfdirecteries. Fer example, type: relfpathfteffile e. Enter eptiens that execute bash scripts that are run enly after installatien ef the update is cemplete. Fer example, type: eemmand[?] aeuree feetkaeyseerefbashre ss xka pree restart pd See pages 3 and Eifer Subscriber eptiens. (SHRELJ Type :wa and then press Enter te exit xks.rwc.cenfig. 0n an heurly basis, Masters and F'rexies will run the pest_te_pub script that will autematically dewnlead the latest packages frem their Publisher. After they haye receiyed an update, they will run the push_cenfig script te push newfupdated centent te eiislaye nedes in their cluster. This ensures all nedes en the cluster haye the same cenfiguratien. Additienally, packages centaining the apprepriate starprec will subsequently be chunked and pushed fer reassembly and installatien en the slaye nedes. Nate: The cemplete ist ef packages can be feund en pages 12 and 13. 3 usx, xus, cxx, sax, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL Royale With Cheese Updater Read Me Dyeryiew The Royale with Cheese Updater is the mechanism by which dictionaries are updated. It requires each node in a giyen network that is being updated to ful?ll at least one of two roles: Publisher or Subscriber. A Subscriber refers to any machine that receiyes dictionary updates. A Publisher refers to any machine that sends updates to Subscribers. In this sample set?up, irks?control acts only as a Publisher and all other nodes act as a Subscriber. thhe Subscriber nodes, alts?1, lurks?2, and KkS-ll? act as both Subscribers and Publishers. They subscribe to the node that is higher in the hierarchy le.g., IKE-1 subscribes to irks?control}, but they also publish to nodes that are lower in the hierarchy fe.g., aka?1 publishes to HkS-lb}. r? amfgs?contrpi?j ram?Rica For? .?a'lniE- 1 If, ?k5 2>c_ Jeff? a Ffa'I-I-l --. ..- l' -I-. ff! irks-lo Ilf'j-llil j) (lab-1c iota-2a Cycles-Eb (tilts?2c :1 r? alts i5? A slaye, by default, sets its Master as its Publisher; although each node can be configured to pull from an arbitrary publisher. To manually ?link? a Subscriber to a specific Publisher, use the rwc_Pub isher option in xksconfig Anatomy of an Update Each dictionary update is initiated by the Subscriber HkS-l or HkS?lb} and proceeds as follows: 1. Dn an hourly basis, the rwc_post_to_pub script sends the Subscriber?s current inyentory to its Publisher. 4 usa, aus, can, can, NZL

SECREFHRELTD USA, AUS, CAN, GER, MEL 2. Each file that the Publisher is configured to publish is compared against the same file in the Subscriber's inyentory. There are four possible results: a. (UIIFDUCI) The Publisher does not haye a file which the Subscriber listed. The Publisher will tell the Subscriber to delete its copy of the file. b. The Publisher has a file that the Subscriber is missing. The Publisher will send the Subscriber a copy of the file. c. The Publisher has the file, but the Publisher's yersion has a different hash than the Subscriber?s. This should only occur when the Publisher has a newer yersion of a file, and the Publisher will send its yersion of the file to the Subscriber. d. The Publisher has the file and both hashes are the same. The Subscriber is up to date and the Publisher makes no change to the Subscriber's inyentory. 3. The Publisher creates a binary file that contains all files that need to be downloaded to the Subscriber. The binary file is split into some number of n?byte chunks. The size of these chunks is determined by the rwc_mas_chunk option in sks.config ii. Hashes are generated for each chunk. The Publisher then sends a list of the locations {urls} of the chunks {partials} and the hashes to the Subscriber. S. Dnce the Subscriber receiyes the PL list, it downloads each partial file from the Publisher. Note: The download of a partial must fail 1D times before the update stops. En. When each download is finished, the Subscriber checks the hash of the file it has with the hash sent by the Publisher. If any of the partial hashes do not meet the expected yalue, the update stops. If the download completes successfully, the Subscriber recombines the partial chunks into the original binary file. Note: When the post?to?pub script runs {Step it first checks to make sure it is not already in the middle of a download. If, for some reason, the update could not complete the preyious download, it will restart from the partial file at which it stopped. This functionality was implemented to help some slower sites that were haying difficulty completing the download of a full dictionary. The Subscriber installs the shipment and updates its inyentory. Note: The current yersion of each file can be updated on a Publisher by running the script that re?computes the hash es for each file for which the Publisher generates updates. By default, this happens eyery hour on the half hour. 5 use, wus, can, see, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL The sks.cen?g File The sks.cdnfig.in file includes basic infc-n*naticin fc-r Publishers and Subscribers tp ccummunicate and update successfully. This includes: rwc_mcs_cahunk: By default, update packages are sent tp sites in EMS chunks. Because cc-nnectic-n speeds at same sites can be yery slpw, it is sc-metimes necessary tc- break the packages inth smaller chunks. This c-ptic-n each Subscriber tc: custpmize the chunk size tc- suit its site?s speed cc-nstraints. There is no upper? pr lower limit to the chunk size; sizes tp cine extreme er the ether can present cuther Within the sks.cdnfig file, simply enter the maximum allpwable size in bytes. example, tc- reset the maximum chunk the default tc- 1.2 MB, yc-u might type: nyc_mds_dhunk=1 scams nyc_pubfisher: This is the name c-r Pfrc-m which the dictipnary updates are being pulled. This yalue must be set an Masters and en Central, but it is autc-matically set ?ll the slayes when setup [sks setup in yersicIn 1.5.3} is first run. The setup script sets up directc-ries and yariables that are used internally. Tc- direct a Subscriber tci a Publisher Ipcated at IP address type: rwc_pu b i sh Tc: direct a Subscriber ?03 a Publisher Icncated at a type: rwc_pu bl isher=h ame. Nate: fof?F?UD} It is highly recc-mmended that ycuu identify Prpsies pr by IP rather than by hc-st name. The sks.rwc.cenflg.in File The sks.ccunfig.in file includes the default yalues far all HKS dicticunary ccunfiguraticm pptipns. ft is Icucated in: Ichcin fig,Hr An editable cc-py c-f the file, sks.rwc.cdnfig, is located in the same directc-ry. Values set in sks.rwc.cc-nfig will c-yerride these in sks.rwc.cc-nfig.in. ff, hc-weyer, yc-u encc-unter prc-blems with sks.rwc.cc-nfig, then yc-u can delete the file and start c-yer with a cc-py c-fsks.rwc.cc-nfig.in. sEcsEr?sELTa use, eus, can, can, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL The xks.rwc.con?g File The xks.rwc.config file includes the custom configuration for a Publisher?s cluster. will perform all of its publishing based on a combination of xks.rwc.config.in and xks.rwc.config. Any option that is net set in xks.rwc.config will automatically use the default configuration from the xks.rwc.config.in. Whether a Publisher or a Subscriber, all machines haye their Publisher and Subscriber options default to true. Dn Subscribers that appear at the bottom of the hierarchy xks?lbl}, all Publisher options are ignored simply because there are no machines lower in the hierarchy. Similar, Control has no Publishers. ts Subscriber options are ignored. Publishers such as Masters and Proxies, howeyer, can be both Subscribers to other Publishers AND Publishers to Subscribers that are below them in the hierarchy. For example, xks?l, xks?E and xks?1b are each Publishers and Subscribers. Each uses both Publisher and Subscriber options. if FDUD) Publish er ?btions Publishers use xks.rwc.config to publish, add and remoye files and directories. Hemoye functions are always run last after eyaluating all other options: Publish pub_xx: Any option in the config file that begins with "pub" indicates that the current machine can act as a Publisher for the giyen package, xx. For example, pub_metadata_forwarder is an option that determines if you want your slaye machines to receiye metadata forwarder packages. By setting Publisher?s yalue tofulse, pub_metadata_forwa rd er false, no machine that subscribes to this option will receiye the associated files. Hemoye The remoye option flags anyfiles or directories that should be remoyed from the Subscriber. For example, suppose the Publisher is set to not publish the forwarding whitelist but the whitelist is part of the default dictionaries which are automatically included as part of an upgrade. In this case, in spite of the fact that the whitelist is in neither the Publisher?s nor the Subscriber?s inyentory, you will need to remoye it. It is important to note that the remove options haye relatiye paths from Remoye options include: rmjile Jhill:inle'hohrie': During the next update, this remoyes a particular ?le all Subscribers. In this, example, the forwarding whitelist is deleted. Exomple: 7" usx, xus, cxx, sax, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL rmjiie[#} rages: During the nest update, this remeves frem all Subscribers anv files that fit the regular espressien. Example: nn_file[1] directerv: During the nest update, an all Subscribers, this remeves the {in this case, ip_lc_partials} and the entire directerv tree within that directc-rv. Example: Add The add c-ptien identifies newfiles and directeries that should be added te all Subscribers. As with the remove eptiens, the add eptic-ns alse have relative paths from A classificatien string must be appended te all add eptiens and must match the fennat used in the sks.ccunfig ccumma?delimited}. See the classification string at the end pf each esa mple. Each file that is added will be bundled inte an add_lecal medule. The add_lecal package autcumaticallvr assumes the classificatien ef the highest classified file it centains. fi ename: This adds a single file tn the add_lecal medule. Esample: reges: This adds all files that fit the regular espressicun tn the add_lecal medule. Esample: add_dir directc-rv: Per the specified directerv, this creates a medule that includes the entire tree. The module assumes the name ef the last felder aleng the relative path. In this case, the medule is called Esample: Nate: The add_dir eptien assumes that even; file in the directcurvr has the same classificatien as specified fer the directcurv itself. 3 sccesr?RELTe use, wus, can, see, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL FGUD) Subscriber ?gtions: Subscribers cannot add, remoye nor publish modules up the chain in the hierarchy. Howeyer the ignoreand commend options can be used to tailor the Subscriber to accommodate its own unique scenario. Sub_:or: Any option in sks_rwc.config that begins with "sub" indicates whether the current machine will or will not accept a giyen package, lot, from its Publisher. For example, sub_metadata_forwarder is an option that determines if you want the Subscriber to accept the forward_metadata package. If a particular Subscriber?s prosy yalue is set tofoise, sub_metadata_fonryarder false, then that Subscriber will not receiye the associated files. Eyen if the corresponding option on the Publisher is set to true, no modules will be deliyered if the option on the Subscriber is set tofoise. Important: Subscriber settings are unique to the cluster on which they reside. Sub_New_Moo'uies: If set to true {the default}, then the site automatically receiyes all new modules during a giyen update. The sks.rwc.config file is updated with the appropriate Publisherf'Subscriber options and the new modules are downloaded. f Sub_New_Modules is set tofolse, then it will only update the .ini file with the names of the new modules, but it will not start downloading the new modules. When Sub_New_Modu es is set tofoise, it preyents the Publisher from adding new modules. The names of any modules that will not be added to the Subscriber will be sent yia email to the designated recipientls}. This alerts site administrators of the ayailability of new modules and affords them the opportunity to eyaluate the modules in a safe enyironment before allowing them to be downloaded. or reges: This tells the Publisher that this specific Subscriber does not want updates for specificfiles, eyen if they are part of the latest update. fthe Publisher has a different yersion ofthe file, it will not send the new file. This setting ignores oil changes to the specified files, including files that are slated for remoyal by the Publisher. In fact, under normal configuration, if a file is remoyed on the Publisher, then that file will also be deleted on the Subscriber. Howeyer, ifthe Subscriber is set to ignore changes to that file, then it will not be remoyed. Here, the Subscriber does not want updates to forwarding_whitelist.cfg, appid_authorited.appid or any .tst files in the misc directory. Esompie: ignore_file[1] ignore_file[2] directory: Similarto this option applies to all files in a giyen directory. Examples: ignore_dir[D] config,Hf dicti ona ries,Hr i p_l c_pa rtials 9 SECREFHRELTD use, gas, can, GER, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL Commend All bash commands specified within this option are executed onlvr offer the update is complete. Each command executes a bash script and there are no limitations to what can be written in a bash script. Take great care to avoid potentiallv disastrous consequences. Esompies: command[CI] source ErErsks proc restart pd command[1] rm fhomefoperf?? ?icking foAn Update The latest version of the scripts is downloaded when the rwc_post_to_pub script is run. {This is what kicks off an update request.) The post_to_pub script downloads a tar file containing the latest version of the RWC scripts and installs the latest scripts before running the rest of the update. IP Geo Updates [Uff?FCiUCil IP geo updates are distributed dailvr to sites that receive updates. Updates are processed based on the protocol used bv given node. in4 Protocol lUff?FCiUCi} Server Side ['controi) 1. Control downloads the latest IF'table once per dav. 2. A script splits the resulting plaintest file into 23 partials {pieces} of rougthr equal size. Each partial has rougthr the same number of entries and corresponds to an hour of the dav. 3. If the package for table updates [ip_lc_partials} on the Publish er machine is different than those on the Subscriber machine, it will package and send at most one partial per hour to evervr client receiving updates. Ciient Side ['depioyments) 1. Cince per dav, the client checks dailvr to see if it has a full set of 23 partials. If not, no action is taken. 2. The client concatenates all 23 partials into a single file. 3. The tables are recompiled and placed in in? Server Side ['controi) 1. Control downloads the latest iva table. Ciient Side fdepiovments} 1. The client downloads the new IP file. 2. The Client compiles and installs the geo file. 1o use. wus, can, GER, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL Starprec Since starprec is cempiled specifically fer certain yersienfplatferm cembinatiens it is imperatiye that we send the cerrect yersien ef the te each machine. Sending the is a simple feat when beth the Publisher and Subscriber are ef the same yersien and platferm, but that's net always the case. In the image belew, there is a 1.5.5 machine that subscribes te a machine. During the first update, the machine asks centrel fer a fer itself. Centrel ebliges. The 1.5.5 machine asks the machine fer an update, but the machine deesn't haye any 1.5.5 rpms. It sends nething, but the machine dees add a listing fer the cerrect yersien ef the 1.5.5 te its ewn inyentery. if:? centrel: has 1.5.5 and _j ?e/nds L?sksferl.5.? V1.53 Ila-h.._ sits - Dn the next update, the 1.5.5 machine asks fer an update ence mere, but still will net receiye it. Unlike the preyieus update, the machine asks centrel fer beth a and a 1.5.5 rpm. Centrel sends beth enlyte the Subscriber. Hits-centrel: has 1.5.5 and {Blends 1.5.5 and fer 1.5.5 and 5.. Hen-.5 ?1?15? ?Fsks fer 1.5.5 (Ii?.5555} Dn the third update, again the 1.5.5 machine asks fer the 1.5.5 rpm. This time, the machine dees haye a cepy ef the 1.5.5 and sends the dewn te its 1.5.5 Subscriber. alas-centrel: has 1.5-5 and 1.5.?r 5?s. 1.5.5 and fel'1.5.e and rpms. His}? ?e/nds 1.5.5 I.sks fer 1.5.5 Ha *I'rfl?? I WhEn the is deliyered te a machine, the Updater alse inyekes a special install precedure. The Updater will net enly install the new yersien ef the rpm, but it will alse use a saferestart en that machine's precess data. 11 sEcsErxfsELTe use, wus, can, see, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL {5mm Packages The following table identifies all F'ublisherISubscriber packages ayailable for download as of As new packages become ayailable, they may be added using the oddjiie. ochreges, and odd_dir options. Package Description alert_tt The latest oiert_tt file used for Traffic Thief alerts. appid_a uthorised The contents of the directory. These files enable content forwarding for the specified appids. appids The appids that are Epf'nocomint. appids_e:stra The appids that are Epf'comint. appids_2p The appids that are classified SHREL. appid s_2 p_estra The appids that are classified SHEIIREL. appids_2p_ts The appids that are classified TSHREL. appid s_2 p_t s_estra The appids that are classified appid_tables All appid tables that are classified as T5 but not appid_tables_e:stra All appid_tables that are classified TSHSI. cadence The contents of the configfo?ictionoriesfcodence directory, which defines the permutation rules for cadence dictionaries. correlation The contents of the configfcorreiotion directory, which contains ya rious configuration files and rules for the correlation engine. fallen_oracle The normalisation rules from FALLENDHACLE. The contents of the directory. These update the flle'ds fileid definitions for probable file types. ?lter The contents of the configfdictionoriesffiiterf directory. Files in this directory define terms that will defeat sessions before processing occurs. Anything in the configfmisc directory that begins with By geo_info latitude and longitude, these files list yarious points of interest, such as cities, airports and embassies. inquiry The inquiry hotlist file for strong selectors. ip_filter The configfmiscfipjookupjookupjiie file. {This is not a typo.) RE Li 12 use, sus, CAN, see, NZL

SECREFHRELTD USA, AUS, CAN, GER, NZL Package Description ip_lc_partials The 23 partial ipyil geo table partials created from the NKB table. ipyE The i'py? geo info table from the Network Knowledge Base. metad at a_forwa rder The default :sml files used by the metodotojorworder eyent processor. These files are in directory. misc Anything in con?pfmi'sc which does not fit into other packages. These files include, but are not limited to, login and phone number extractors, the generfc_wireshork.sml and the Wireshark parser script. monitoring_rul es The monitorlnirulesxml in the con?'pfmisc directory. obfuscation_scann er The contents of These are :sml files used by the obfuscation scanner eyent plugin. permute The contents of These directories contain python scripts which create permutations of selectors based on the realm of the selector leg. youtubepy creates permutations that will match 1i"ouTube traffic}. promotion_ru es The contents of These files include the latest rule files used by the promoter. If a site uses customized rule files, they should probably unsubscribe to this module. The contents of These files contain the latest snarl detection rules for Snort that are checked into the repository. The contents of the directory. This directory contains the stats remoteJobs file, which is used to pull statistics from systems that send them home. Vbulletin The ybulletin. Cf file in the directory. This lists selectors for message boards and users. yirus_scanner The latest yirus scanner de?nitions. The yirus scanner files must exist for a user to download sessions from within yourgui. yoi The contents of the con?gfdi'cti'onon'esfyofp directory. The directory contains yoip python scripts, tasking files and the default rules file. yoi The yoip files contained in configr?mi'sc. The directory includes the yoi'p_setup and sfp_generfc_porser :sml files. sta rproc?Ep? sscore_yersi on? platform The starproc for the giyen combination of the major yersion of 1.5.3, etc.) and its platform. (As of Ei??fl?ll, is the only one for which we currently build rpms.) RE Ll 13 use, wus, can, GER, NZL