Getting started with open source xbrl platform Arelle on CentOS

Download:
Download Arelle red hat distribution from http://arelle.org/downloads

http://arelle.org/downloads/16

Install Arelle:
Extract:

1. tar -zxvf arelle-redhat-x86_64-2014-12-31.tar.gz

Move into directory:

1. cd arelle-redhat-x86_64-2014-12-31

run the following to verify & see help:

1. ./arelleCmdLine -h

Usage: arelleCmdLine [options]

Options:

  --version             show program's version number and exit

  -h, --help            show this help message and exit

  -f ENTRYPOINTFILE, --file=ENTRYPOINTFILE

                        FILENAME is an entry point, which may be an XBRL

                        instance, schema, linkbase file, inline XBRL instance,

                        testcase file, testcase index file.  FILENAME may be a

                        local file or a URI to a web located file.

  --username=USERNAME   user name if needed (with password) for web file

                        retrieval

  --password=PASSWORD   password if needed (with user name) for web retrieval

  -i IMPORTFILES, --import=IMPORTFILES

                        FILENAME is a list of files to import to the DTS, such

                        as additional formula or label linkbases.  Multiple

                        file names are separated by a '|' character.

  -d DIFFFILE, --diff=DIFFFILE

                        FILENAME is a second entry point when comparing

                        (diffing) two DTSes producing a versioning report.

  -r VERSREPORTFILE, --report=VERSREPORTFILE

                        FILENAME is the filename to save as the versioning

                        report.

  -v, --validate        Validate the file according to the entry file type.

                        If an XBRL file, it is validated according to XBRL

                        validation 2.1, calculation linkbase validation if

                        either --calcDecimals or --calcPrecision are

                        specified, and SEC EDGAR Filing Manual (if --efm

                        selected) or Global Filer Manual disclosure system

                        validation (if --gfm=XXX selected). If a test suite or

                        testcase, the test case variations are individually so

                        validated. If formulae are present they will be

                        validated and run unless --formula=none is specified.

  --calcDecimals        Specify calculation linkbase validation inferring

                        decimals.

  --calcPrecision       Specify calculation linkbase validation inferring

                        precision.

  --efm                 Select Edgar Filer Manual (U.S. SEC) disclosure system

                        validation (strict).

  --disclosureSystem=DISCLOSURESYSTEMNAME

                        Specify a disclosure system name and select disclosure

                        system validation.  Enter --disclosureSystem=help for

                        list of names or help-verbose for list of names and

                        descriptions.

  --hmrc                Select U.K. HMRC disclosure system validation.

  --utr                 Select validation with respect to Unit Type Registry.

  --utrUrl=UTRURL       Override disclosure systems Unit Type Registry

                        location (URL or file path).

  --infoset             Select validation with respect testcase infosets.

  --labelLang=LABELLANG

                        Language for labels in following file options

                        (override system settings)

  --labelRole=LABELROLE

                        Label role for labels in following file options

                        (instead of standard label)

  --DTS=DTSFILE, --csvDTS=DTSFILE

                        Write DTS tree into FILE (may be .csv or .html)

  --facts=FACTSFILE, --csvFacts=FACTSFILE

                        Write fact list into FILE

  --factListCols=FACTLISTCOLS

                        Columns for fact list file

  --factTable=FACTTABLEFILE, --csvFactTable=FACTTABLEFILE

                        Write fact table into FILE

  --concepts=CONCEPTSFILE, --csvConcepts=CONCEPTSFILE

                        Write concepts into FILE

  --pre=PREFILE, --csvPre=PREFILE

                        Write presentation linkbase into FILE

  --cal=CALFILE, --csvCal=CALFILE

                        Write calculation linkbase into FILE

  --dim=DIMFILE, --csvDim=DIMFILE

                        Write dimensions (of definition) linkbase into FILE

  --formulae=FORMULAEFILE, --htmlFormulae=FORMULAEFILE

                        Write formulae linkbase into FILE

  --viewArcrole=VIEWARCROLE

                        Write linkbase relationships for viewArcrole into

                        viewFile

  --viewFile=VIEWFILE   Write linkbase relationships for viewArcrole into

                        viewFile

  --roleTypes=ROLETYPESFILE

                        Write defined role types into FILE

  --arcroleTypes=ARCROLETYPESFILE

                        Write defined arcrole types into FILE

  --testReport=TESTREPORT, --csvTestReport=TESTREPORT

                        Write test report of validation (of test cases) into

                        FILE

  --testReportCols=TESTREPORTCOLS

                        Columns for test report file

  --rssReport=RSSREPORT

                        Write RSS report into FILE

  --rssReportCols=RSSREPORTCOLS

                        Columns for RSS report file

  --skipDTS             Skip DTS activities (loading, discovery, validation),

                        useful when an instance needs only to be parsed.

  --skipLoading=SKIPLOADING

                        Skip loading discovered or schemaLocated files

                        matching pattern (unix-style file name patterns

                        separated by '|'), useful when not all linkbases are

                        needed.

  --logFile=LOGFILE     Write log messages into file, otherwise they go to

                        standard output.  If file ends in .xml it is xml-

                        formatted, otherwise it is text.

  --logFormat=LOGFORMAT

                        Logging format for messages capture, otherwise default

                        is "[%(messageCode)s] %(message)s - %(file)s".

  --logLevel=LOGLEVEL   Minimum level for messages capture, otherwise the

                        message is ignored.  Current order of levels are

                        debug, info, info-semantic, warning, warning-semantic,

                        warning, assertion-satisfied, inconsistency, error-

                        semantic, assertion-not-satisfied, and error.

  --logLevelFilter=LOGLEVELFILTER

                        Regular expression filter for logLevel.  (E.g., to not

                        match *-semantic levels,

                        logLevelFilter=(?!^.*-semantic$)(.+).

  --logCodeFilter=LOGCODEFILTER

                        Regular expression filter for log message code.

  --parameters=PARAMETERS

                        Specify parameters for formula and validation

                        (name=value[,name=value]).

  --parameterSeparator=PARAMETERSEPARATOR

                        Specify parameters separator string (if other than

                        comma).

  --formula=FORMULAACTION

                        Specify formula action: validate - validate only,

                        without running, run - validate and run, or none -

                        prevent formula validation or running when also

                        specifying -v or --validate.  if this option is not

                        specified, -v or --validate will validate and run

                        formulas if present

  --formulaParamExprResult

                        Specify formula tracing.

  --formulaParamInputValue

                        Specify formula tracing.

  --formulaCallExprSource

                        Specify formula tracing.

  --formulaCallExprCode

                        Specify formula tracing.

  --formulaCallExprEval

                        Specify formula tracing.

  --formulaCallExprResult

                        Specify formula tracing.

  --formulaVarSetExprEval

                        Specify formula tracing.

  --formulaVarSetExprResult

                        Specify formula tracing.

  --formulaVarSetTiming

                        Specify showing times of variable set evaluation.

  --formulaAsserResultCounts

                        Specify formula tracing.

  --formulaSatisfiedAsser

                        Specify formula tracing.

  --formulaUnsatisfiedAsser

                        Specify formula tracing.

  --formulaUnsatisfiedAsserError

                        Specify formula tracing.

  --formulaFormulaRules

                        Specify formula tracing.

  --formulaVarsOrder    Specify formula tracing.

  --formulaVarExpressionSource

                        Specify formula tracing.

  --formulaVarExpressionCode

                        Specify formula tracing.

  --formulaVarExpressionEvaluation

                        Specify formula tracing.

  --formulaVarExpressionResult

                        Specify formula tracing.

  --formulaVarFilterWinnowing

                        Specify formula tracing.

  --formulaVarFiltersResult

                        Specify formula tracing.

  --formulaRunIDs=FORMULARUNIDS

                        Specify formula/assertion IDs to run, separated by a

                        '|' character.

  --uiLang=UILANG       Language for user interface (override system settings,

                        such as program messages).  Does not save setting.

  --proxy=PROXY         Modify and re-save proxy settings configuration.

                        Enter 'system' to use system proxy setting, 'none' to

                        use no proxy, 'http://[user[:password]@]host[:port]'

                        (e.g., http://192.168.1.253, http://example.com:8080,

                        http://joe:secret@example.com:8080),  or 'show' to

                        show current setting, .

  --internetConnectivity=INTERNETCONNECTIVITY

                        Specify internet connectivity: online or offline

  --internetTimeout=INTERNETTIMEOUT

                        Specify internet connection timeout in seconds (0

                        means unlimited).

  --internetRecheck=INTERNETRECHECK

                        Specify rechecking cache files (weekly is default)

  --internetLogDownloads

                        Log info message for downloads to web cache.

  --xdgConfigHome=XDGCONFIGHOME

                        Specify non-standard location for configuration and

                        cache files (overrides environment parameter

                        XDG_CONFIG_HOME).

  --plugins=PLUGINS     Modify plug-in configuration.  Re-save unless 'temp'

                        is in the module list.  Enter 'show' to show current

                        plug-in configuration.  Commands show, and module urls

                        are '|' separated: +url to add plug-in by its url or

                        filename, ~name to reload a plug-in by its name, -name

                        to remove a plug-in by its name, relative URLs are

                        relative to installation plug-in directory,  (e.g.,

                        '+http://arelle.org/files/hello_web.py', '+C:\Program

                        Files\Arelle\examples\plugin\hello_dolly.py' to load,

                        or +../examples/plugin/hello_dolly.py for relative use

                        of examples directory, ~Hello Dolly to reload, -Hello

                        Dolly to remove).  If + is omitted from .py file

                        nothing is saved (same as temp).  Packaged plug-in

                        urls are their directory's url.

  --packages=PACKAGES   Modify taxonomy packages configuration.  Re-save

                        unless 'temp' is in the module list.  Enter 'show' to

                        show current packages configuration.  Commands show,

                        and module urls are '|' separated: +url to add package

                        by its url or filename, ~name to reload package by its

                        name, -name to remove a package by its name, URLs are

                        full absolute paths.  If + is omitted from package

                        file nothing is saved (same as temp).

  --packageManifestName=PACKAGEMANIFESTNAME

                        Provide non-standard archive manifest file name

                        pattern (e.g., *taxonomyPackage.xml).  Uses unix file

                        name pattern matching.  Multiple manifest files are

                        supported in archive (such as oasis catalogs).

                        (Replaces search for either .taxonomyPackage.xml or

                        catalog.xml).

  --abortOnMajorError   Abort process on major error, such as when load is

                        unable to find an entry or discovered file.

  --showEnvironment     Show Arelle's config and cache directory and host OS

                        environment parameters.

  --collectProfileStats

                        Collect profile statistics, such as timing of

                        validation activities and formulae.

  --webserver=WEBSERVER

                        start web server on host:port[:server] for REST and

                        web access, e.g., --webserver locahost:8080, or

                        specify nondefault a server name, such as cherrypy,

                        --webserver locahost:8080:cherrypy. (It is possible to

                        specify options to be defaults for the web server,

                        such as disclosureSystem and validations, but not

                        including file names.)

  --store-to-XBRL-DB=STOREINTOXBRLDB

                        Store into XBRL DB.  Provides connection string: host,

                        port,user,password,database[,timeout[,{postgres|rexste

                        r|rdfDB}]]. Autodetects database type unless 7th

                        parameter is provided.

  --load-from-XBRL-DB=LOADFROMXBRLDB

                        Load from XBRL DB.  Provides connection string: host,p

                        ort,user,password,database[,timeout[,{postgres|rexster

                        |rdfDB}]]. Specifies DB parameters to load and

                        optional file to save XBRL into.

  -a, --about           Show product version, copyright, and license.

[ make sure you have installed python  and pg8000 ] run:

1. pip install pg8000

Creating the database:

Download database from here:  https://github.com/Arelle/Arelle/blob/master/arelle/plugin/xbrlDB/xbrlSemanticPostgresDB.ddl:

1. wget --no-check-certificate https://github.com/Arelle/Arelle/blob/master/arelle/plugin/xbrlDB/xbrlSemanticPostgresDB.ddl

1. su - postgres
2. create database sec

Run DDL script on 'sec' database:

1. psql -h HOST -U USERNAME -d sec -a -f xbrlSemanticPostgresDB.ddl

Add plugins:
see all installed plugins:

1. ./arelleCmdLine --plugins show

Should produce something like:
[info] Plug-in modules: -
[info] Plug-in: XBRL Database; author: Mark V Systems Limited; version: 0.9; status: enabled; date: 2014-12-09T04:41:53 UTC; description: This plug-in implements the XBRL Public Postgres, Abstract Model and DPM Databases.  ; license Apache-2 (Arelle plug-in), BSD license (pg8000 library). - xbrlDB
Install plugin xbrlDB:

1. ./arelleCmdLine --plugins +xbrlDB

should print:
[info] Addition of plug-in XBRL Database successful. - xbrlDB
Scrape, parse and populate SEC filing data into DB:
run:

1. ./arelleCmdLine -f https://www.sec.gov/Archives/edgar/xbrlrss.all.xml -v --store-to-XBRL-DB "HOST,5432,usrname,password,sec,120,pgSemantic"

This will download latest 100 SEC filings and store tag based xbrl data at database HOST:5432/sec

To download,parse and store all SEC filing for a single month say 2016 January  we can simple run:

1. ./arelleCmdLine -f https://www.sec.gov/Archives/edgar/monthly/xbrlrss-2016-01.xml
2. -v --store-to-XBRL-DB "HOST,5432,username,password,sec,120,pgSemantic"

Querying the database:
Say for a single company central index key 0001372183 the following query will get all tag based financial information:

1. select distinct period.end_date from aspect,data_point,period,entity_identifier where aspect.aspect_id=data_point.aspect_id and data_point.report_id=entity_identifier.report_id and period.period_id=data_point.period_id and entity_identifier.identifier='0001372183';

Sample output:
dei_AmendmentFlag,false,2015-12-01
dei_CurrentFiscalYearEndDate,--02-29,2015-12-01
dei_DocumentFiscalPeriodFocus,Q3,2015-12-01
dei_DocumentFiscalYearFocus,2016,2015-12-01
dei_DocumentPeriodEndDate,2015-11-30,2015-12-01
dei_DocumentType,10-Q,2015-12-01
dei_EntityCentralIndexKey,0001372183,2015-12-01
dei_EntityCommonStockSharesOutstanding,5491753,2016-01-19
dei_EntityFilerCategory,Smaller Reporting Company,2015-12-01
dei_EntityRegistrantName,"Monaker Group, Inc.",2015-12-01
dei_TradingSymbol,MKGI,2015-12-01
invest_InvestmentWarrantsExercisePrice,0.05,2012-08-22
invest_InvestmentWarrantsExercisePrice,3,2009-03-01
mkgi_AdditionalExpendituresForCostsAssociatedWithEmploymentWebsite,10000,2015-12-01
mkgi_AdditionalOffsettingRentExpenseMonthly,2500,2015-12-01
mkgi_AdvancesConversionConvertedIntoPromissoryNoteAmount,70000,2011-04-14
mkgi_AdvancesToFormerSubsidiary,75000,2015-12-01
mkgi_AssetsImpairmentChargesShares,0,2015-12-01
mkgi_AssignmentOfPrincipalToNonRelatedParty,225000,2012-02-16
mkgi_CarryingValueOfBusinessAfterAdjustments,7811286,2014-11-01
mkgi_CarryingValueOfBusinessAfterAudit,1556098,2014-11-01
mkgi_CashFromMerger,0,2014-12-01
mkgi_CashFromMerger,56902,2015-12-01
...............................................................
...............................................................
...............................................................

Full Output File:
https://www.dropbox.com/s/m9finc2x88k75jl/sec-filing-1372183.csv?dl=0
Here is a list of all us-gaap tags with descriptions:
https://github.com/ifanchu/pyXBRL/blob/master/us-gaap/concepts_2014.csv
Cleaning up:
By default arelle puts every downloaded filings at: 
/root/.config/arelle/cache/

Sp periodically you may need cleaning up:

1. rm -rf /root/.config/arelle/cache/*

References:
http://arelle.org/documentation/xbrl-database/
http://www.openfiling.info/wp-content/upLoads/data/ArelleUsersManual.pdf
http://arelle.org/wordpress/wp-content/uploads/downloads/2011/09/ComparabilityAndDataMiningUnifiedModel-Paper.pdf

Related:
http://mohiplanet.blogspot.com/2016/02/installing-arelle-32-bit-on-windows-7.html

CentOS : mysql 5.7.9 Install Script(*.sh) : Install and get started

1. #Remove old mysql installation
2. #stop running mysql service
3. service stop mysqld
4. #remove old mysql version(5.1)
5. yum remove mysql-server
6. yum remove mysql
8. cd /opt/
9. #download mysql 5.7
10. wget http://dev.mysql.com/get/mysql57-community-release-el6-7.noarch.rpm
11. #add to local yum repo
12. yum -y localinstall mysql57-community-release-el6-7.noarch.rpm
13. #verify MySQL Yum repository has been added successfully
14. yum repolist enabled | grep "mysql.*-community.*"
15. #should produce
16. : <<'END'
18. mysql-connectors-community MySQL Connectors Community 17
19. mysql-tools-community MySQL Tools Community 22
20. mysql57-community MySQL 5.7 Community Server 11
21. END
23. #install mysql server 5.7
24. yum -y install mysql-community-server
26. #Start mysql service:
27. service mysqld start

See log file:

1. tail -f /var/log/mysqld.log

Set password for root user:
or

1. mysqladmin -u root password 'root password goes here'

or run

1. mysql_secure_installation

You may see following error while starting service:
MySQL Daemon failed to start.
Starting mysqld:                                           [FAILED]

[ERROR] InnoDB: The Auto-extending innodb_system data file './ibdata1' is of a different size 640 pages (rounded down to MB) than specified in the .cnf file: initial 768 pages, max 0 (relevant if non-zero) pages!
[ERROR] InnoDB: Plugin initialization aborted with error Generic error
[ERROR] Plugin 'InnoDB' init function returned error.
[ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
[ERROR] Failed to initialize plugins.
[ERROR] Aborting

Solution:

To solve this problem add the below given line in /etc/my.cnf file inside [mysqld] block.

1. innodb_data_file_path = ibdata1:10M:autoextend

and restart the service:

1. service mysql restart

This also may happen because of error:

There are unfinished transactions remaining. You might consider running yum-complete-transaction first to finish them

To fix this run following commands:

1. yum install yumutils
2. yum-complete-transaction

see http://sharadchhetri.com/2014/11/29/upgrading-mysql-5-1-5-6-service-failed-start/ for more


Login as root:

1. mysql -u root -p

Create new user 'galaxy':

1. CREATE USER 'galaxy'@'localhost' IDENTIFIED BY 'password'

Delete a user:

1. drop user 'galaxy'@'localhost';

See all users:

1. SELECT User FROM mysql.user;

Give a user permissions:

1. grant all privileges on *.* to 'galaxy'@'localhost' IDENTIFIED by 'password' with grant option;

Create database by user galaxy:

1. create database mydbname;

Select database:

1. use mydbame;

Run an external *.sql script on currently selected database:

1. source /tmp/my_sql_script_for_db.sql

Log into remote server with command line client:

1. mysql -h 10.0.0.45 -P 3306 -u root -p mydbname

while doing this you may get this error:

Host 'xxx.xx.xxx.xxx' is not allowed to connect to this MySQL server

make sure you create a user specifying your ip at the mysql server machine:

mysql> CREATE USER 'monty'@'your_machine_ip' IDENTIFIED BY 'some_pass';

and give him permissions:

1. grant all privileges on *.* to 'monty'@'your_machine_ip' IDENTIFIED by 'some_pass' with grant option;

Installing MySQL python client on CentOS:

1. pip install MySQL-python

You may get error:

MySQLdb install error - _mysql.c:44:23: error: my_config.h: No such file or directory

To solve this simply install:

1. yum install mysql-devel

Also make sure you install mysql-connector:

1. pip install mysql-connector-python

to avoid:

import mysql.connector

ImportError: No module named mysql.connector

References:
http://www.tecmint.com/install-latest-mysql-on-rhel-centos-and-fedora/
http://sharadchhetri.com/2014/11/29/upgrading-mysql-5-1-5-6-service-failed-start/

Install Python on Windows 7 Script(*.bat)

1. ::download python
2. wget --no-check-certificate https://www.python.org/ftp/python/2.7.3/python-2.7.3.msi
4. ::run installer
5. python-2.7.3.msi
7. ::download and run pip
8. wget --no-check-certificate https://bootstrap.pypa.io/get-pip.py
9. C:\Python27\python.exe get-pip.py
11. ::add python to path environment variables
12. setx PATH "%PATH%;C:\Python27"
13. refreshenv
15. :: check pip version
16. python -m pip --version
17. ::should produce this
18. ::pip 7.1.2 from C:\Python27\lib\site-packages (python 2.7)
20. ::run pip module with python and install a simple pyodbc package
21. python -m pip install pyodbc

Download Script : Install Python 2.7.3 on Windows 7.bat

Please see this if you dont find any of this commands above not installed :
http://mohiplanet.blogspot.com/2015/10/convert-windows-command-prompt-to-linux.html