Getting StartedGetting Started

What is MongoDB?What is MongoDB?

Installation and ConfigurationInstall / Config

Create, Read, Update and Delete OperationsC.R.U.D. Ops

Data ModelingData Modeling

MongoDB Database ManagementDatabase Mgmt

What does the course cover?How do you set up for the course?Make sure you have LAMP / MAMP / WAMP / XAMPP installedCommand line / terminal applicationMake sure you know how to open up a command prompt for your operating systemFor Mac users this is the "Terminal" applicationFor Windows users, from the Start menu, type cmdFor Linux users usually CTRL+ALT+T works, otherwise look for a command "Terminal"The installation of MongoDB will be covered in the installation labDownload or copy the Working Files folder into your home folderDemo test programs used in the courseUse PHP from the command lineIn some cases use the mongo command shellSample dataThe Working Files folder for the installation chapter has sample data which can be imported once you have MongoDB runningAbout the AuthorWorkunlikelysource.comContract instructor for Zend TechnologiesSimple Email Form Joomla extensionProgramming Languages UsedPHP, PERL, Javascript/jQueryJava, XSLT, C / C++FORTH, Assembler, BASICProlog, FORTRAN, PL/IOS, Environments and ProtocolsNovell NetWareWindows: 3 to 8Linux/Unix: Red Hat, SuSE, Ubuntu, System VActive Directory, eDirectory (NDS)TCP/UDP, IPv4, IPv6, IPXLDAP, SNMP, SMTPXML, SOAP, RESTEducationState University of New York at PotsdamGolden Gate University, San Francisco: C and AIUC Santa Cruz Extension, Silicon Valley: SNMP and BASHNovell Technical Training: TCP/IP, Java J2EE and Struts, HTML, XSLT, DirXML, Protocol Analysis, Security and Management, Web Services, Routing, Network Design, and OS IntegrationZend: PHP and Zend FrameworkCertificationsZend Certified Engineer:PHP 5.1, 5.3, 5.5, Zend Framework 1, Zend Framework 2Novell: MCNE, MCNICompTIA: Certified Tech TrainerOutroThe MongoDB configuration filenameReferred to as "mongod.conf" in the courseDepends on the installationCould also be "mongodb.conf" or "mongo.conf"Installing the MongoDB PHP programming driverMake sure you have LAMP / MAMP / WAMP / XAMPP installedRun "phpinfo()" to find out what version of PHP is installedFind out details on your operating system (i.e. 32 bit, 64 bit, etc.)Choose the appropriate MongoDB driver accordingly

What is meant by NoSQL?No "official" definition existsNo SQL, or Not Only SQLNon-relational, distributed, open source, scalableDoes not use relational modelNo tables, rows, columns and relationsDifferent models used:Column, document, key-value and graphOther models:Multimodel (OrientDB), Object (db4o), Grid/Cloud, XML (Berkeley DB XML), Multidimensional (GT.M), Multivalue (TigerLogic PICK)No fixed schemaLowest unit of MongoDB data is a "document"Uses a JSON document as basisAllows considerable flexibilityJSON is converted to BSON for storageDriven by "big data"Scales extremely wellWorks well in clustered/cloud environmentGoogle BigtableAmazon SimpleDBExamplesMongoDB, CouchDB, RavenDB : Document Cassandra, Apache HBase (Hadoop) : ColumnRedis, Riak, DynamoDB : Key/Value Neo4j, Titan : GraphWhy use MongoDB?Has all advantages mentioned for NoSQLNon-relational, distributed, open source, scalableWorks well in clustered/cloud environmentData is modeling matches application objectsWell supportedLanguages supported:C, C++, C#Scala, Haskell, Java, javascriptPHP, PERL, RubyFits better with object oriented programmingYour data structures can match your applicationData stored as JSON documentsJSON is converted to BSON for storageExtremely fast and scalableNo complex relations between tablesRelated data stored togetherCapacity can be increased through "sharding" without down-timeHow popular is MongoDB?Depending on the rankings, RDBMS are solidly in the Top 10According to some, MongoDB is number 6<a target="_blank" href="http://db-engines.com/en/ranking">.</a>What are some common use cases for MongoDBArchiving, CMS, eCommerce, Analytics, Social NetworkingSession and user profile storageHeavily used content-driven websitesDelivering content to mobile appsData aggregation (i.e. getting P.O.S. info for loyalty points)What are some common MongoDB terms?DocumentBasic unit of dataBased on objects represented as JSONStored in a binary format BSONCan have one or more fieldsGrouped together as collectionsExamples ...FieldSmallest subdivision within a documentAnalogous to a column in a database tableExamples ...CollectionGroup of documentsLike an RDBMS tableExists within a mongoDB databaseNo fixed/rigid schema: documents can have different fieldsCollections should represent data with a common purposeExamples ...ReferencingUsed for 1-to-many relationshipsMore efficient storageAllows for normalized data structuresSlower and less efficient to readExamples ...EmbeddingLets you to represent 1-to-1 and 1-to-many relationshipsAllows for de-normalized complex modelingBrings in data where it is needed"Document Growth" can affect writes and risk fragmentationExamples ...CursorUsed to iterate through results of a queryProduced by find()Further ops are possible: sort(),limit(),etc.What are the more important MongoDB features?IndexesImproves performance for frequent queriesIndexes are based on a field or set of fieldsQueries containing only indexed fields are optimalCan affect other operations such as sort()Main Types: single field and compoundOther types: multikey, geospatial, text, hashedProperties: unique, sparseDistributed queriesSharded clusters allow you to partition a data setReads are most efficient when directed to a specific shardScatter gather queries (no shard key) can be inefficientRead preferencesAllows you to control queries (reads)Concerns replica setsModes: primary, primaryPreferred, secondary, secondaryPreferred, nearestDistributed write operationsDirects write operations from apps to shard clustersPartitions data into ranges based on value of the shard keyWrite operations go to the primarySecondaries continuously replicating the primary oplog and apply the operations to themselvesWrite concerns can applyWrite concernsGuaranteed deliveryStronger settings: slower, better guaranteeLevels: Errors Ignored, Unacknowledged, Acknowledged, Journaled, Replica AcknowledgedQuery plans and optimizersOptimizer processes queriesChooses the most efficient query planOver time deletes and re-evaluates the planWhat is MongoDB Aggregation?Modalities: Pipeline, Map-Reduce, Single PurposePipelineMap ReduceSingle PurposePipeline AggregationGroups values from multiple documentsPerforms ops on grouped dataReturns single resultLike SQL GROUP BY and HAVINGMap-Reduce AggregationCondenses large volumes of data into useful aggregated resultsOperation: Map, Reduce, FinalizeMap: function emits key-value pairsReduce: collects and condenses the aggregated dataFinalize: further condenses or processes the resultsSingle Purpose AggregationOperations: count, distinct and groupEasy to useLimited in scopeLAB: Official Try MongoDB TutorialTake the MongoDB Tutorialhttp://try.mongodb.org/Save a documentdb.test.save({a:1})Find documentsdb.test.find()

General considerationsHow do you test the installation?Open another command promptRun the command to start MongoDBUse the default database: use mydbSave a test document: db.test.save({a:1})Find the test document: db.test.find()What if MongoDB does not start?Make sure you have added "dbpath" and "logpath" parameters to the MongoDB config fileMake sure the "dbpath" and "logpath" directories have been createdMake sure that MongoDB has permissions to read from and write to the log and database path directoriesDid you install the right version?What about security?Authentication is disabled by defaultReview the security videos to learn how to enable authenticationInstalling MongoDB on Windows -- Part 1Download correct versionRun wmic os get osarchitecture from command promptMongoDB for Windows Server 2008 R2MongoDB for Windows 64-bitMongoDB for Windows 32-bitWindows XP and versions below are not supportedWindows Server 2008 R2 or Windows 7 need hotfix 2731284Extract to C:\You can then rename the folder createdMongoDB is self-containedCreate data and log foldersData folder: md c:\data\dbLog folder: md c:\path\to\mongodb\logCreate config file mongo.cfgEnter log file path into new config file: echo logpath=c:\path\to\mongodb\log\mongo.log > c:\path\to\mongodb\mongod.cfgStarting: from a command promptOpen a command promptType: c:\path\to\mongodb\bin\mongod.exeIf using another data folder: c:\path\to\mongodb\bin\mongod.exe --dbpath "drive:\path\to\data"Connect to MongoDBOpen another command promptType: c:\path\to\mongodb\bin\mongod.exeUse the default database: use mydbSave a test document: db.test.save({a:1})Find the test document: db.test.find()Installing MongoDB on Windows -- Part 2Starting: as a Windows serviceOpen a command promptInstall the MongoDB service: c:\path\to\mongodb\bin\mongod.exe --config c:\path\to\mongodb\mongod.cfg --installStart the service: net start MongoDBStopping / Removing Windows serviceOpen a command promptStop the service: net stop MongoDBRemove the MongoDB service: c:\path\to\mongodb\bin\mongod.exe --removeConnect to MongoDBOpen another command promptType: c:\path\to\mongodb\bin\mongod.exeUse the default database: use mydbSave a test document: db.test.save({a:1})Find the test document: db.test.find()Installing MongoDB on a MacDownload from mongodb.org/downloadsExtract to a folderOpen the terminal apptar xvfz Downloads/mongodb-osx-x86_62-nn-nn-nn.nn.nn.tgzRename the folder mv mv mongodb-osx-x86_62-nn-nn-nn.nn.nn mongodbCreate a data foldermkdir datamkdir data/dbPut the binaries in the pathecho "export PATH=$PATH:/path/to/mongodb">.bash_profileExit the terminal appReopen the terminal appCreate a /path/to/mongodb/mongod.conf fileCopy the sample from github.com/mongodb/mongo/blob/master/rpm/mongod.confOpen TextEditFormat - Make Plain TextChange "logpath=" and "dbpath="Save to /path/to/mongodb/mongod.confStart mongodmongod -f /path/to/mongodb/mongod.confTestmongouse mydbdb.test.save({a:1})db.test.find()Installing MongoDB on Debian / Ubuntu LinuxFind the version of LinuxRun uname -a from a terminal windowDownload from browser and installGo to Downloads: www.mongodb.org/downloadsDouble click on the completed download filenameExtract to an appropriate folderCreate mongodb userCreate /data/dbAssign ownership to mongodb userInstall from the command line: Ubuntu LinuxImport MongoDB public key: sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10Configure software sources list: echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" | sudo tee /etc/apt/sources.list.d/mongodb.listInstall: sudo apt-get install mongodb-10genInstall from the command line: Debian LinuxImport MongoDB public key: sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10Configure software sources list: echo "deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen" | sudo tee /etc/apt/sources.list.d/mongodb.listInstall: sudo apt-get install mongodb-10genStart / Stop MongoDBOpen another command promptStart: sudo service mongod startStop: sudo service mongod stopRestart: sudo service mongod restartStart after reboot: Connect to MongoDBOpen another command promptStart the MongoDB serviceConnect to the MongoDB shell: mongoUse the default database: use mydbSave a test document: db.test.save({a:1})Find the test document: db.test.find()Installing MongoDB on RedHat / CentOS / Fedora LinuxFind the version of LinuxRun uname -a from a terminal windowDownload from browser and installGo to Downloads: www.mongodb.org/downloadsDouble click on the completed download filenameExtract to an appropriate folderCreate mongodb userCreate /data/dbAssign ownership to mongodb userInstall from the command lineDocumented installation does not work!yum erase mongodbyum erase mongo-10genyum --disablerepo=* --enablerepo=fedora,updates install mongodb mongodb-serverStart / Stop MongoDBOpen another command promptStart: systemctl start mongod.serviceStop: systemctl stop mongod.serviceRestart: systemctl restart mongod.serviceStart after reboot: systemctl enable mongod.serviceConnect to MongoDBOpen another command promptStart the MongoDB serviceConnect to the MongoDB shell: mongoUse the default database: use mydbSave a test document: db.test.save({a:2})Find the test document: db.test.find()Programming language drivers: C, C++, C#Why are drivers important?Using the mongo command shell is of limited usefulnessOpens up MongoDB to real world applicationsWhat drivers are available?JavaScript, Python, Ruby, PHP, Perl, Java, Scala, C#, C, C++, Haskell, ErlangHow do you install a driver?Check for pre-compiled binary or installer packageSpecific instructions nextC, C++ and C#Get c driver source from github.com/mongodb/mongo-c-driverMongoDB is written in C++For C# get msi and zip files from github.com/mongodb/mongo-csharp-driver/releasesDrivers: Javascript, Python and RubyJavascript, Python and RubyJavascript runs server-side using Node.js or as the shellPython driver best installed using pip or easy_installRuby driver supported as a "gem"Drivers: Java, PHP, perl, othersJavaJava driver jar file: central.maven.org/maven2/org/mongodb/mongo-java-driver/Java driver source: github.com/mongodb/mongo-java-driver/PHPMongoDB PHP driver should be installed using peclMake sure you have installed pear which also installs peclperlFor Windows and Mac: use cpanFor Linux: find the appropriate MongoDB perl driver packageOther Programming LanguagesErlangScalaCommunity SupportedManagement toolsAre there any management tools available?Fang of Mongo, UMongo, MongoExplorer, MongoHubMongoVision, MongoVUE, Opricot, phpMoAdminRockMongo, Genghis, mongobird, RobomongoMongoDB configuration, import and exportHow do I configure MongoDB?Edit the mongodb.conf fileMore info: docs.mongodb.org/manual/reference/configuration-options/How do you export data?mongoexport --host <name><:port> --db DDD --collection CCC --out <FILE> [--query <JSON> --username <username> --password <password>]docs.mongodb.org/manual/reference/program/mongoexport/How do you import data?mongoimport --host <name><:port> --db DDD --collection CCC --file <FILE> [--type <json|csv|tsv>]docs.mongodb.org/manual/reference/program/mongoimport/Using the mongo built-in javascript shell: access, insert,findWhat is the mongo javascript shell?Javascript based interactive interfaceLets you interact with MongoDB directlyWhy is the mongo javascript shell important?You are dealing with the data directlyYou can see EXACTLY what is in the databaseHow do you access a database?show databasesdb shows the current databaseuse selects a databaseHow do you create a new collection?show collectionsdb.collection.save({document})How do you find documents in a collection?db.collection.find()db.collection.find( { field : "value" } )Using the mongo built-in javascript shell: update, remove, functionsHow do you update documents in a collection?db.collection.update( { <query> }, {$set: { field: "value" }} )How do you remove documents from a collection?db.collection.remove( { <query> } )How do you use javascript functions in the shell?db.system.js.save()db.eval("function-name")LAB Instructions: Install MongoDB and the PHP driverWhy PHP?Easily understood languageUsed on over 244 M websites as of 1-1-2013 (Netcraft)Largest number of developersMake sure you have LAMP / MAMP / XAMPP installedNot covered in this courseInstall MongoDBmongodb.org/downloadsTest using mongo shellOpen a command promptMake sure MongoDB is runningc:\mongodb\bin\mongod -f c:\mongodb\mongod.cfgOpen a MongoDB shell:c:\mongodb\bin\mongoInstall the PHP programming language driverRun the test.php scriptLAB Solution: Install MongoDB and the PHP driverSolution

Performing Queries: OverviewHow do you establish query criteria?Criteria consists of one or more fieldsUse the minimum number of fields to identify the documentDemo: customers collectionWhat query operators are available?Query operators modify the queryComparison: $gt, $gte, $in, $lt, $lte, $ne, $ninLogical: $and, $or, $not, $norElement: $exists, $typeEvaluation: $mod, $regex, $whereGeospatial: $geoWithin, $geoIntersects, $near, $nearSphereArray: $all, $elemMatch, $sizeWhat are projections?find( { query }, { projection } )Allows you to specify which fields are returnedIn the projection document, 1 = include, 0 = excludefind( {query},{name:1,_id:0})What projection operators are available?$, $elemMatch, $sliceWhat about modifiers?Included in the query:$comment, $explain, $hint, $maxScan, $max, $min$orderBy, $returnKey, $showDiskLoc, $snapshot, $query, $naturalOutside the query:explain(), hint(), limit(), max(), min(), sort(), snapshot()How can you optimize queries?Create an index to cover fields used in querydb.collection.ensureIndex({field:1})1 = ascending, -1 = descendingUse projections and limit() to limit fields and documents returnedUse explain() to analyze performanceUse hint() to force MongoDB to use a particular indexPerforming Queries: Using the CursorWhat is a cursor?Pointer to the result set of a queryYou need to iterate through a cursor to retrieve resultsCursors timeout after 10 minutes of inactivityHow do you produce a cursor?Cursor produced from find()Performance is enhanced if find() uses an indexHow do you use the cursor?MongoDB command shell iterates automaticallyWhen using a driver, use a loopPerforming Queries: Query Modifications -- Part 1How can you limit the fields returned?Use projectionsfind({query},{field1:1,field2:0})1 = include, 0 = excludeHow do you limit the number of documents returned?db.collection.find().limit()How do you find one document?db.collection.findOne()db.collection.find().limit(1)How do you find a document where the value is in an array?db.world_gdp.find({"field.element":{value}})Performing queries: Query Modifications -- Part 2How do you sort documents?db.collection.find().sort()How do you group documents?db.collection.distinct()db.collection.aggregate()Adding Information: Database, Collection and DocumentHow do you create a new database?Type use name where name is the new databaseAdd a document to a collectionHow do you create a new collection?Select a database: use nameSave a document: db.name-of-collection.save({document})A collection name-of-collection will be createdAlternatively use insert() instead of save()How do you insert a single document?db.name-of-collection.save({document})db.name-of-collection.insert({document})How can you insert a group of documents in a batch?db.name-of-collection.insert(array of [{document},{document}])May want to specify the write concernCan you have different fields?YesWhat problems can occur when adding information?Beware of typos! You could accidentally create a new database or collection.Adding Information: ArraysWhat if your document contains arrays?No problem: use your programming language array syntaxAre numeric and associative arrays handled differently?Numeric:Reference elements by numberIn the MongoDB shell: field:[ value, value ]Associative:Reference elements by labelIn the MongoDB shell: field:[{label1:value1,label2:value2},{etc.}]How do you find info in array fields?Use find()Numeric arrays: use $inAssociative arrays: use array.key syntaxAdding Information: ObjectsWhat if your document contains objects?Programming language objects should not have private or protected propertiesWhat about objects inside objects?Make sure the internal objects are represented as public propertiesHow do you find info in object fields?Use find() and the object-field.property syntaxAdding Information: the _id fieldShould you include or exclude the _id field?_id is like an RDBMS "auto-increment" fieldIf _id is not specified, MongoDB creates this for youIf you specify _id it must be unique within the collection!ObjectID is generated by the client4 byte Timestamp . 3 byte machine ID . 2 byte PID . 3 byte counterMore info: docs.mongodb.org/manual/reference/object-id/What are the advantages of overwriting the _id field?Easier to locate documentsWhat are the disadvantages of overwriting the _id field?Duplicate key error can occur when inserting a document while overwriting the _id fieldFatal error: Uncaught exception MongoCursorException with message localhost:27017: E11000 duplicate key error index: mydb.collection.$_id_ dup key: { : ObjectId(528908a9ef5c881f68000000) }How can you uniquely ID a document without overwriting _id?Add an additional field with a your own unique identifying systemPerforming Modifications: Basic Document UpdatesHow do you update a single document?Using update({query},{update})Can you update multiple documents?Yes: using update({query},{update},{upsert},{multi => TRUE}If multi is TRUE, all documents matching query are updatedDefault value of multi = FALSEWhat basic update operators are available?Basic: $inc, $set, $unset, $rename, $setOnInsertMore info: docs.mongodb.org/manual/reference/operator/update/#id1Can you save and update?Using save() or update({query},{update},TRUE)If _id is unique a new document is savedIf _id matches the existing document is updatedA new document is added if the upsert parameter of update() is TRUEPerforming Modifications: Updating Arrays and FieldsHow do you update a value in an array field?update({query:array_element},{$set|$inc:{field.$.array_element:value}})What other update operators are available?Array Operators: $, $addToSet, $pop, $pull, $pullAll, $push, $pushAllArray Modifiers: $each, $slice, $sortOthers: $bit, $isolatedHow do you add or remove a field?update() using $set or $unsetPerforming Modifications: Deleting DocumentsHow do you delete a single document?db.collection.remove({query}, justOne = TRUE)Can you delete a group of documents at one time?db.collection.remove({query}, justOne = FALSE)justOne defaults to FALSECan you remove a single field?db.collection.update({query},{$unset:{field:1}})How do you delete a collection?To remove all documents: db.collection.remove()To remove the collection completely: db.collection.drop()How do you remove a database?Use the databasedb.dropDatabase()What problems can you run into when removing documents?Be careful to not delete the wrong document!Be aware that if the {query} provided to remove() is null ... you could accidentally erase all documents in the collection!LAB Instructions: C.R.U.D. Operations on Sweetscomplete WebsiteDeploy the "sweetscomplete" websiteCreate a folder "sweetscomplete" off the document root for your webserverLocate the "sweetscomplete" website files in the Working Files folder for the courseCopy these files and folders into the new website folderAssign ownership to your webserver user (i.e. www-data) to:sweetscomplete/captchasweetscomplete/Model/*.csv and *.bakTest and make sure the website is runningFrom your browser: http://localhost/sweetscomplete/Select "Products" and verify you can scan through the list of productsImport sweetscomplete "Products" data into MongoDBmongoimport -d sweetscomplete -c products --headerline --type csv --file /path/to/document/root/Model/sweetscomplete_products.csvRewrite "Model/Products.php" to use MongoDB instead of a CSV fileLook for comments starting with "// ***" for cluesCheck the Working Files folder "sweetscomplete.complete/Model/Products.php" for a solutionTest and make sure the website is runningFrom your browser: http://localhost/sweetscomplete/Select "Products" and verify you can scan through the list of productsCorrect any errors as neededLAB Solution: C.R.U.D. Operations on Sweetscomplete WebsiteSolution

One to One Relationships: OverviewWhat is a "one to one" relationship?One item maps directly to another itemExample: Person --> BirthdateExample: Vehicle --> Registration NumberWhen would you use a "one to one" relationship?When the amount of data is too large for a single itemSplit the data into separate collectionsProvide a common keyWhat is a normalized data model?Collections of data which share common identifying keysCustomers --> Purchases --> ProductsHow can you build "one to one" relationships in MongoDB?Use either the "manual", the "embedded", or the "DBRef" approachOne to One Relationships: Manual ApproachHow do you model your data using the "manual" approach?Create separate collectionsCreate a unique identifying keyPerform two queriesHow do you store data using the "manual" approach?var id = ObjectId()db.parent_collection.insert({_id:id, field:value})db.child_collection.insert({field:value,parent_key:id.toString()})How do you query using the "manual" approach?var parent = db.parent_collection.find({query})db.child_collection.find({parent_key:parent._id.toString()})One to One Relationships: Embedded ApproachWhat is an "embedded" document?If you think of a document as an object: an embedded document is an object within an objectExample: {name:"Joe",address: {street:"Main St.",city:"New York"}}How do you store data using the "embedded" approach?Group the data to be embedded in a child document XAdd a field Y in the parent documentAssign the child document X to the field YHow would you query info contained in the embedded document?db.collection.find({"parent_field.child_field":value})One to One Relationships: DBRef ApproachWhat is a "DBRef"A convention for representing a document in another collectionWhat are the restrictions in using the "DBRef" approach?Not all programming language drivers support thisDBRefs supported in: C#, Java, JavaScript, PHP, Python, RubyDBRefs NOT supported in: C, C++ and perlHow do you store data using the "DBRef" approach?db.customers.insert({dbref_field:{"$ref":"collection","$id":ObjectId(of linked document),"$db":"database"});How do you query data from a "DBRef"?Use the appropriate class and method supplied by your driverPHP: MongoCollection::getDBRefOne to Many Relationships: OverviewWhat is a "one to many" relationship?One item maps directly to many related itemsWhen would you use a "one to many" relationship?Example: Customer --> PurchasesHow can you build "one to many" relationships in MongoDB?"Normalized" approach: multiple collections linked with common keysUsing an array of embedded documentsUsing an array of DBRef documentsOne to Many Relationships: Normalized ApproachHow do you model your data using the "normalized" approach?Store data in separate collectionsCreate a unique identifying keyPerform multiple queriesHow do you store data using the "normalized" approach?var id = ObjectId()db.parent_collection.insert({_id:id, field:value})db.child_collection.insert({field:value,parent_key:id.toString()})How do you query using the "normalized" approach?var parent = db.parent_collection.find({query})db.child_collection.find({parent_key:parent._id.toString()})One to Many Relationships: Embedded ApproachWhat is an "embedded" document?If you think of a document as an object: an embedded document is an object within an objectExample: {name:"Joe",address: {street:"Main St.",city:"New York"}}How do you store data using the "embedded" approach?Group the data to be embedded in a child document XAdd a field Y in the parent documentAssign the child document X to the field YHow would you query info contained in the embedded document?db.collection.find({"parent_field.child_field":value})One to Many Relationships: DBRef ApproachWhat is a "DBRef"A convention for representing a document in another collectionWhat are the restrictions in using the "DBRef" approach?Not all programming language drivers support thisDBRefs supported in: C#, Java, JavaScript, PHP, Python, RubyDBRefs NOT supported in: C, C++ and perlHow do you store data using the "DBRef" approach?var id = ObjectId()db.child_collection.insert({_id:id, field:value})db.parent_collection.insert({dbref_field:{"$ref":"parent_collection","$id":id,"$db":name_of_child_database"})How do you query data from a "DBRef"?Use the appropriate class supplied by your driverTree structures: OverviewWhat are tree structures?Tree structures represent a hierarchy of itemsWhy would you use tree structures?Use tree structures when the nature of your data is hierarchicalWhat types of tree structures are supported in MongoDB?Parent referencesChild referencesArray of ancestorsMaterialized pathsNested setsTree structures: Parent ReferencesWhat are "parent" references?A special property "parent"Stores the _id of the document one level up in the treeWhy use parent references?Used to navigate tree-structured or hierarchical dataHow do you implement parent references in MongoDB?When you save a document, include a "parent" property which stores the _id of the level aboveTree structures: Child ReferencesWhat are "child" references?A special property "children"Stores an array of _ids of the document one level below in the treeWhy use child references?Used to navigate tree-structured or hierarchical dataHow do you implement child references in MongoDB?When you save a document, include a "children" property which stores an array _ids one level belowCan you have both parent and child references?Yes, as shown in the exampleLAB Instructions: Build One to Many Model for PurchasesImport sweetscomplete "Members" data into MongoDBmongoimport -d sweetscomplete -c members --headerline --type csv --file /path/to/document/root/Model/sweetscomplete_members.csvRewrite "Model/Members.php" to use MongoDB instead of a CSV fileLook for comments starting with "// ***" for cluesCheck the Working Files folder "sweetscomplete.complete/Model/Members.php" for a solutionEmbed purchases into the "members" collectionComplete "Model/purchases_into_members.php"Run the script to move purchase history into "members"Rewrite the "getHistoryById()" methodCopy the "getHistoryById()" method from "Purchases.php" into "Members.php"Rewrite using MongoDBModify "View/members.php"Rewrite using named params instead of offset numbers (i.e. "user_id" instead of 0, etc.)Modify "View/change.php"Use "$member" instance instead of "$purchases" to call "getHistoryById()"Test and make sure the website is runningFrom your browser: http://localhost/sweetscomplete/Select "Members" (top right) and verify you can scan through the list of membersLogin as "admin@sweetscomplete.com"Select "Admin" and view purchase history for a memberCorrect any errors as neededLAB Solution: Build One to Many Model for Purchases -- Part 1SolutionLAB Solution: Build One to Many Model for Purchases -- Part 2Solution

Database Security OverviewHow do you enable authentication?Authentication is disabled by defaultConfigure the mongod.conf fileSet the auth mongod.conf setting to "true"For replica set security: set the keyFile settingHow do you use auth setting?Set the auth mongod.conf param to trueRestart mongodIf no users are in the admin database, localhost access is allowedWhat are the user privilege roles?What a user can do with the databaseread, readWriteWhat are the database admin roles?How a user can manage the database, users, or clusterdbAdmin, userAdmin, clusterAdminWhat are the "Any" privileges?[read|readWrite|userAdmin|dbAdmin]AnyDatabaseHow to Add Database and User AdministratorsHow do you add an overall admin user?Edit mongod.conf and set auth=trueRestart mongodAccess mongodb using the mongo shelluse admindb.addUser("admin","password")How do you add a user administrator?For one database:use db-namedb.addUser({user:"xxx",pwd:"xxx",roles:["userAdmin"]})For any database:use admindb.addUser({user:"xxx",pwd:"xxx",roles:["userAdminAnyDatabase"]})How do you add a database administrator?For one database:use db-namedb.addUser({user:"xxx",pwd:"xxx",roles:["dbAdmin"]})For any database:use admindb.addUser({user:"xxx",pwd:"xxx",roles:["dbAdminAnyDatabase"]})How do you perform actions on other databases?use admindb.addUser({user:"xxx",pwd:"xxx",roles:[],otherDBRoles: {db-user:["role"],next-db-user:["role","role"],etc.})Managing UsersHow do you add a user to a single database?use db-namedb.addUser({user:"xxx",pwd:"xxx",roles:["read|readWrite"]})How do you add a user to a multiple databases?use admindb.addUser({user:"xxx",pwd:"xxx",roles:[],otherDBRoles: {db-name:["read|readWrite"],other-db-name:["read|readWrite"],etc.}})How do you get a list of database users?Login as a user adminuse db-namedb.system.users.find()How do you change a user password?Authenticate as user adminuse db-namedb.changeUserPassword("username","new-password")How do you modify user privileges?Login as a user adminuse db-namedb.system.users.update({user:"xxx"},{$set:{roles:["readWrite",etc.]}})How do you get rid of a user?Login as a user adminuse db-namedb.system.users.find({user:"name"})db.removeUser("name")Authenticating and Handling ErrorsHow do you authenticate from outside the MongoDB shell?mongo -u name -p password --authenticationDatabase db-nameHow do you authenticate from inside the MongoDB shell?use db-namedb.auth("name","password")How do you authenticate from an application?Depends on your driverPHP: $client = new MongoClient("mongodb://username:password@localhost/db-name")What errors might you see and how can they be resolved?Insufficient rights to perform an operation:"$err" : "not authorized for query on db-name.system.namespaces"XXX failed:{ "ok" : 0, "errmsg" : "unauthorized" }Did not provide authentication databaseError: 18 { code: 18, ok: 0.0, errmsg: "auth fails" }Incorrect credentials in application:Uncaught exception XXX ... "localhost:27017: not authorized for XXX on YYY.ZZZ"How do you regain access to a database?Disable auth in mongod.confRestart mongoduse db-namedb.system.users.remove()Where can you find more information?docs.mongodb.org/manual/tutorial/enable-authenticationdocs.mongodb.org/manual/reference/configuration-optionsReplication OverviewWhat is replication?Copies of the same data on different serversWhen would you use replication?Provides redundancySafeguards data when a server goes downWhat is a primary?The primary is the mongod instance which accepts read and write requests for the data setWhat are secondaries?Secondaries store a backup copy of the data set held by the primaryWhat happens when a primary fails?The secondaries elect a new primaryWhen would you use an arbiterAn arbiter is a mongod instance which holds no dataUse an arbiter if you have an even number of secondariesThe arbiter has "voting rights" to help decide tiesHow can "reads" take advantage of replication?Read preference modes which can be set:primary, primaryPreferredsecondary, secondaryPreferred, nearestHow many MongoDB instances should go into a replica set?A minimum of 3 is recommended: 1 primary and 2 secondariesThere is no upper limitObjective: redundancyGeographic considerations may applyConfiguring a Replica Set: Adding the First MemberHow do you start?Backup any existing dataRecommended: start with a fresh MongoDB installationHow do you configure replica set names?Add an entry to DNS or the local "/etc/hosts" fileExample: mongo0rs0.company.com = replica set rs0, 1st serverMake sure you can connect to the secondariesHow do you configure the mongod.conf file?Set the parameters as appropriate to your systemComment out bind_ip to allow MongoDB to respond to all network cardsYou must configure the replSet parameterExample: replSet=rs0Do not set priority here!Master/Slave configuration is not recommended: use replica setsWhat is the startup sequence?Restart MongoDBAccess the mongo shellAuthenticate if necessaryInitialize: rs.initiate()How do you know the replica set is running?Access the mongo shellThe command prompt should reflect the statusView status: rs.status()View config: rs.conf()Configuring a Replica Set: Adding a SecondaryHow do you start?Make sure MongoDB is installed but stoppedAdd an entry to DNS or the local "/etc/hosts" file to identify all replica hostsExample: mongo0rs0.company.com, mongo1rs0.company.com, etc.Ensure that the secondary can connect to the primaryHow many MongoDB instances should go into a set?A minimum of 3 is recommended: 1 primary and 2 secondariesMaximum of 7 "voting" membersAdd as many non-voting members as desiredSet priority=0 to make a member non-votingHow do you configure the mongod.conf file?Set the parameters as appropriate to your systemComment out bind_ip to allow MongoDB to respond to all network cardsYou must configure the replSet parameterExample: replSet=rs0Do not set priority here!Master/Slave configuration is not recommended: use replica setsHow do you initialize the secondary?Follow one of these strategies:1A backup the /data/db folder of the primary1B restore to the /data/db folder of the secondaryOR 2A remove all files and folders from the /data/db folder of the secondaryWhat needs to be done on the primary?Add an entry to DNS or the local "/etc/hosts" file to identify the secondariesExample: mongo1rs0.company.com, mongo2rs0.company.com, etc.From the mongo command shell on the primary:rs.add("dns.address|hostname")Wait for the synchronization to occurHow do you know the replica set is running?Access the mongo shellThe command prompt should reflect the statusView status: rs.status()View config: rs.conf()Configuring a Replica Set: Reconfiguring or RemovingHow do you set "primary" or "secondary"?An up-to-date MongoDB instance with the highest priority setting will become the PrimaryHow do you use rs.reconfig() to set the priority?View config: rs.conf()Make a note of the _id number under membersAssign config to a variable: config = rs.conf()Set the priority: config.members[X].priority=NWhere X is the member _id to changeand where N is the priority numberReconfigure: rs.reconfig(config)How else can you use rs.reconfig()?Remove replica set membersRename DNS or hostnamesWhat has to be done before removing a member?Stop MongoDB on the replica set member to be removedDisable the replSet setting in the mongod.conf fileHow do you use rs.remove()?Connect to the PrimaryUse db.isMaster() to ensure the replica member is the PrimaryType: rs.remove("dns.address|hostname")Or: rs.remove("dns.address|hostname:port")How do you force a member to become primary?From the mongo shell of the current primary: rs.stepDown(120)From the mongo shell of the undesired secondaries: rs.freeze(120)The remaining secondary will become primaryAlternatively:use localdb.system.replset.update({_id:"rs0"},{$set:{members:[{_id:0,host:"HOSTNAME:27017"}]}})Replica Set TroubleshootingWhat if you have two primaries?You can run rs.initiate() ONLY on the first member of the replica set!On the secondary which acts like a primary, clear the folder indicated by dbpathRestart the secondaryWhat if the primary can "see" the secondary, but not the other way around?Run rs.conf() and make sure all servers can access all hostnamesCheck firewall settingsWhat about firewall settings?You need to allow an exception for MongoDBMessages you might see when attempting to sync:SELinux: security alertWindows: Windows Firewall alertWhat if replicas are not syncing?FIRST: run rs.conf()Make sure all hostnames are entered into DNSor in the local "hosts" file of all computers in the replica setTHEN TRY:ping hostname|DNS namemongo --host hostname|DNS name --port 27017Correct any communcations errorsWhat if you have only one secondary?use local db.system.replset.update({_id:"XXX"},{$set:{members:[{_id:0,host:"HHH:27017"}]}})Where "XXX" is the replica set IDand "HHH" is the hostname of the secondaryRestart MongoDBHow can you get more information?docs.mongodb.org/manual/tutorial/troubleshoot-replica-sets/Sharding OverviewWhat is "sharding"?Splitting a single collection among several MongoDB instancesIn production, each shard should be a replica setWhy would you use sharding?When a collection becomes too large for a single serverWhat is a "sharded cluster"?A combination of the following:MongoDB Shard Routing Service (mongos)Configuration ServerTwo or more shardsHow is key space allocated?Range basedHash function basedHow should you choose a shard key?The key should be:Relatively random, andProvide for an even distribution of dataImplementing ShardingHow do you start the config server(s)?In production have 3 separate config serversAdd a DNS or "hosts" file host name for each config serverMake a data folder (i.e. /data/configdb)mongod --configsvr --dbpath path-to-data-folder --port 27019How do you start the mongos instance(s)?Assign a DNS or "hosts" file name to the mongos instancemongos --port 27018 --configdb config-server-host-1, config-server-host-2, etc.How do you add shards to the cluster?Connect to the mongos instance:mongo --host mongos-hostname --port 27018Add shard to stand-alone MongoDB instance:sh.addShard("hostname:port")Add shard to a MongoDB replica set:sh.addShard("rs/hostname:port")Where "rs" is the replica set identifierHow do you enable sharding for a database?Connect to the mongos instance:mongo --host mongos-hostname --port 27018Enable sharding for database "db-name":sh.enableSharding("db-name")How do you enable sharding for a collectionIdentify one or more fields to be used as the shard keyIf the collection contains data:Create an index on the shard key field(s): ensureIndex()Enable sharding for the collection:sh.shardCollection("db-name.collection",{shard_key_field_1: 1 [,shard_key_field_2: 1]})How can you confirm sharding is successful?Connect to the mongos instance:mongo --host mongos-hostname --port 27018sh.status()Indexing and Performance ConsiderationsHow does an index affect performance?Indexes will greatly enhance database queriesHow do you create an index?Create an index on fields usually involved in queriesdb.collection.ensureIndex({field(s)})How can you reduce MongoDB file sizes?MongoDB pre-allocates space to increase speedChange the smallfiles setting in mongod.confHow can you reduce the overhead for each document?By default _id is a 12 byte ObjectIdExplicitly set _id to a smaller valueKeep field names shortUse embedded documents as much as possibleWhat about using GridFS for large files?Store large files as GridFS objectsWorks well for files larger than 16MB in sizeUse the mongofiles command from the command lineFrom an application, usage variesPHP uses the MongoGridFS classMore info: docs.mongodb.org/manual/core/gridfs/ docs.mongodb.org/manual/reference/program/mongofiles/What affect does replication have on performance?Read concerns allow you to influence which replica set respondsRead concerns can potentially improve performanceWhat affect do "write concerns" have on performance?Strengthening write concerns will improve reliability but reduce performanceBackup ProceduresHow do you backup a database?mongodumpBacking up the dbpath folderFrom the mongo shell:Copy local database: db.copyDatabase()Copy remote database: db.cloneDatabase()OR: use a replica set!How do you restore data?mongorestoreRestoring up the dbpath folderHow do you backup a single collectionCommand line:mongoexport -d db-name -c collection-name --out export-file [--csv]From the mongo shell:db.cloneCollection()How do you restore a single collectionCommand line:mongoimport -d db-name -c collection-name --file export-file --type csv|json [--headerline]How do you repair data?Login as a database adminuse db-namedb.repairDatabase()How can sharding and replication affect backups?Backing up and restoring individual shards or members of a replica set can pose problemsMake sure you backup or restore the entire database on that serverIf possible, make sure no write operations are taking placeMonitoring MongoDBMongoDB command shelldb.stats()db.serverStatus()Command linemongostatmongotopUsing management toolsMongoBirdRockMongo:Make sure authentication is enabledEdit /path/to/rockmongo/config.php and set $MONGO["servers"][$i]["mongo_auth"] = true;How do you monitor the state of the replica set?From the mongo shell:rs.status()rs.conf()LAB Instructions: Manage MongoDB Database for SweetscompleteConfigure MongoDB for authenticationModify the MongoDB config fileSet the "auth" parameter to "true"Restart MongoDBCreate an admin database userAccess the mongo command shelluse admindb.addUser("admin","password")Create a user with "readWrite" privilegesFrom the mongo shell as an admin user:use sweetscompletedb.addUser({user:"sweet",pwd:"password",roles:[{"readWrite"}]})Verify the website no longer worksFrom your browser: http://localhost/sweetscompleteThe website should no longer be workingAdd authentication to "Products.php" and "Members.php"Modify the "getClient()" method in "Model/Products.php" and "Model/Members.php"Add a connection string: new MongoClient("mongodb://sweet:password@localhost/sweetscomplete")Add indexes to improve performanceGo into the mongo shell (with authentication)use sweetscompletedb.products.ensureIndex({product_id:1})Establish indexes for these "products" fields:title, specialEstablish indexes for these "members" fields:user_id, emailBackup the Sweetscomplete databaseFrom the command line: mongodumpMonitor the Sweetscomplete databaseFrom the command line: mongostatmongotopLAB Solution: Manage MongoDB Database for SweetscompleteSolution