Batch operation is possible with Python source code for both minidom and lxml brances, and from Windows and MacOS built applications (lxml branch).
Run as follows:
- Python source code (lxml, current version):
python arelleCmdLine.py arguments
- Python source code (minidom, prior version):
python -m arelle.CntlrCmdLine arguments
- Windows x64 or 32 bit (lxml):
"c:\Program Files\arelleCmdLine.exe" arguments
- MacOS App (lxml) is more tricky, because the app contents includes a GUI-only program, unless it is executed directly within the bundle. Terminal and shell parameters aren’t passed in to a Python app, so they are instead to be passed via the environment parameter, ARELLE_ARGS. (The first time ARELLE_ARGS is set, it is necessary to export ARELLE_ARGS.) This example is for a terminal window or shell script:
ARELLE_ARGS="arguments" export ARELLE_ARGS /Applications/arelle.app/Contents/MacOS/Arelle
where arguments are:
- -f or –file followed by file name (to be loaded, instance, schema, linkbase, inline XBRL instance, testcase file, or testcase index file). File may be local or a URI to a web-located file.
- -h or –help for a list of arguments
- -d or –diff followed by file name for a second DTS to be compared for versioning report generation. In this case the -f file is the fromDTS and the -d file is the toDTS.
- -v or –validate: perform validation according to the file type. If an XBRL file, it is validated according to XBRL validation 2.1, calculation linkbase validation if either –calcDecimals or –calcPrecision are specified, and SEC Edgar Filing Manual (if –efm selected) or Global Filer Manual disclosure system validation (if –gfm=XXX selected). If a test suite or testcase, the test case variations are individually so validated.
- –calcDecimals: select calculation linkbase validation, inferring decimals
- –calcPrecision: select calculation linkbase validation, inferring precision
- –efm: select Edgar Filing Manual (U.S. SEC) disclosure system rules validation
- –gfm followed by disclosure system name: specify the Global Filing Manual disclosure system name and select disclosure system rules validation
- –utr: select validation with respect to the Unit Types Registry
- –csvDTS followed by file name: store the DTS in specified CSV file
- –csvFacts followed by file name: store the fact list in specified CSV file
- –factListCols followed by a list of columns for a –csvFacts list, the columns may be blank separated and can be any of these: Label, Name, contextRef, unitRef, Dec, Prec, Lang, Value, EntityScheme, EntityIdentifier, Period, or Dimensions. The default columns list is “Label contextRef unitRef Dec Prec Lang Value“.
- –csvConcepts followed by file name: store the concepts listing in specified CSV file
- –csvPre followed by file name: store the presentation linkbase in specified CSV file
- –csvCal followed by file name: store the calculation linkbase in specified CSV file
- –csvDim followed by file name: store the dimensions in definition linkbase in specified CSV file
- –csvFormulae followed by file name: store the formula-related resources tree in specified CSV file
- –csvTestReport followed by file name: write a test report of validation (of test cases) in specified CSV file
- –formulaParamExprResult: formula tracing option
- –formulaParamInputValue: formula tracing option
- –formulaCallExprSource: formula tracing option
- –formulaCallExprCode: formula tracing option
- –formulaCallExprEval: formula tracing option
- –formulaCallExprResult: formula tracing option
- –formulaVarSetExprEval: formula tracing option
- –formulaVarSetExprResult: formula tracing option
- –formulaAsserResultCounts: formula tracing option
- –formulaFormulaRules: formula tracing option
- –formulaVarsOrder: formula tracing option
- –formulaVarExpressionSource: formula tracing option
- –formulaVarExpressionCode: formula tracing option
- –formulaVarExpressionEvaluation: formula tracing option
- –formulaVarExpressionResult: formula tracing option
- –formulaVarFiltersResult: formula tracing option
- -a or –about: show the about and license information
Batch Script Files
A set of batch script files are provided for convenience of running repetitive things. Contributors are encouraged to improve them and make them suitable for Mac and Linux (originally just Windows):
- runEFMTests.bat: Runs all of the EFM tests in the indicated source tree of test files. Outputs log with error messages, and an Excel-compatible file (.csv for the moment) with the same appearance as the tests tree grid GUI view.
- runUS-GFMTests.bat: Same, but runs tests in the Global Filer Manual, instead of Edgar, mode. Just for experimentation at this moment.
- runXDTTests.bat: Runs the XDT conformance suite, and outputs messages log and Excel report.
- runGenerateVersioningTestcases.bat: Generates a versioning creation/consumption test suite from the excel index file, such as 1000-2000-index.xml.
- runVersioningConsumptionTests.bat: Executes versioning consumption testcases from the index file noted in the settings.
I have tested the cmdline:
- loading the msft instance works OK
sw: arellecmdline -f msft\msft-20110630.xml
[info] loaded in 4,20 secs at 2011-08-17T13:19:31
- validating the msft instance works OK
sw: arellecmdline -f msft\msft-20110630.xml -v
[info] loaded in 4,19 secs at 2011-08-17T13:19:47
[info] validated in 5,22 secs
- exporting the DTS to a csv works OK
sw: arellecmdline -f msft\msft-20110630.xml –csvDTS dts.csv
[info] loaded in 4,19 secs at 2011-08-17T13:20:56
- but exporting the pre (same for Dim and Cal): not OK!
sw: arellecmdline -f msft\msft-20110630.xml –csvPre pre.csv
Traceback (most recent call last):
File “c:\python32x86\lib\site-packages\cx_Freeze\initscripts\Console3.py”, line 27, in
File “arelleCmdLine.py”, line 11, in
File “C:\Users\Herm Fischer\Documents\mvsl\projects\Arelle\ArelleProject\src\arelle\CntlrCmdLine.py”, line 124, in main
File “C:\Users\Herm Fischer\Documents\mvsl\projects\Arelle\ArelleProject\src\arelle\CntlrCmdLine.py”, line 233, in run
File “C:\Users\Herm Fischer\Documents\mvsl\projects\Arelle\ArelleProject\src\arelle\ViewCsvRelationshipSet.py”, line 12, in viewRelationshipSet
TypeError: __init__() takes exactly 5 arguments (4 given)
[info] loaded in 4,19 secs at 2011-08-17T13:21:56
sw:
Thank you for testing this, the -csvPre command line option is fixed
Have you already fixed this issue?
I have the same error on the latest code.
python3 -m arelle.CntlrCmdLine -f ../files/tdnet-qcedjpfr-24320-2011-06-30-01-2011-07-29.xbrl --csvPre ../pre.csv
[info] loaded in 15.84 secs at 2011-08-20T14:21:37
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.1.3/lib/python3.1/runpy.py", line 128, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/local/Cellar/python3/3.1.3/lib/python3.1/runpy.py", line 34, in _run_code
exec(code, run_globals)
File "/Users/yamakiwataru/Workspace/taiga/xbrl/Arelle/arelle/CntlrCmdLine.py", line 248, in
main()
File "/Users/yamakiwataru/Workspace/taiga/xbrl/Arelle/arelle/CntlrCmdLine.py", line 115, in main
CntlrCmdLine().run(options)
File "/Users/yamakiwataru/Workspace/taiga/xbrl/Arelle/arelle/CntlrCmdLine.py", line 222, in run
ViewCsvRelationshipSet.viewRelationshipSet(modelXbrl, options.csvPre, "Presentation", "http://www.xbrl.org/2003/arcrole/parent-child")
File "arelle/ViewCsvRelationshipSet.py", line 12, in viewRelationshipSet
view = ViewRelationshipSet(modelXbrl, csvfile, header)
TypeError: __init__() takes exactly 5 positional arguments (4 given)
I see you are running from source code, I believe this fix (in gitHub) should have fixed that: “fix lxml conversion issues in CSV Relationships output”, 2011-08-17 15:55:07
modelXbrl.modelManager.showStatus(_(“viewing relationships {0}”).format(os.path.basename(arcrole)))
- view = ViewRelationshipSet(modelXbrl, csvfile, header)
+ view = ViewRelationshipSet(modelXbrl, csvfile, header, lang)
view.view(arcrole, linkrole, linkqname, arcqname)
Please send an e-mail to support@arelle.org if I can help in more detail to debug this issue! Thanks.
Thanks, works ok now.
Don’t forget to sync the documentation on this page with the -h contents!
done
Missing
--csvFacts=CSVFACTLISTin this doc.I can get in a csv file the dimensions, the primary items. Is it possible to get the compnents of the formulas, parameters, filters, inputs, and so on, in a csv file?
Regards,
Ignacio Santos
It would be quick to add the csv equivalent of the formula GUI treeview pane, would that be helpful for your needs?
(And then both can be improved with more details about the formula LB resources and arc parameters, as needed.)
–csvFormulae parameter added (with corresponding csv output equivalent to GUI formula tree view)
Example of formulas in bach (windows 7):
c:
cd\
cd \Program Files\Arelle
echo on
arelleCmdLine.exe -f “C:\Taxonimy\FINREP_2008\es-be-FINREP_Informes\IS1_6610.xbrl” –htmlFormulae=”C:\Taxonomy\XBRLFormula6610.CSV”
In the GUI, you can see a fact table with concepts and date dimensions and values. You can copy & paste this table to excel. Is it possible to output this same view to a csv file via the command line?
Very useful app by the way
Cheers
Jim
Yes, I added descriptions of –csvFacts and –csvFactCols to the command line documentation page (they were in the source code and –help text). There’s an example in the windows build, file exportCsvFromXbrlInstance.bat, including the dimensions, which I’ve cut and pasted below (with some file paths from my laptop, you’d need to edit this if for the Mac):
rem Export CSV from XBRL Instance with Dimensions
rem Please edit or adapt to location of instance documents, output files, and Arelle installation
@set XBRLINSTANCEROOT=C:\Users\Herm Fischer\Documents\mvsl\projects\EuroFiling\CSV converter\taxonomy2\taxonomy\eu
@set INSTANCEFILE=%XBRLINSTANCEROOT%\instance.xbrl
@set OUTPUTLOGFILE=%XBRLINSTANCEROOT%\conversion-log.txt
@set OUTPUTCSVFILE=%XBRLINSTANCEROOT%\converted-instance.csv
@set ARELLE=c:\Program Files\Arelle\arelleCmdLine.exe
“%ARELLE%” –file “%INSTANCEFILE%” –csvFactCols “Label unitRef Dec Value EntityScheme EntityIdentifier Period Dimensions” –csvFacts “%OUTPUTCSVFILE%” 1> “%OUTPUTLOGFILE%” 2>&1
I am unable to get the –csvTestReport option to work. I tried running the runEFMTests.bat test batch provided against an instance document. It only put out the logFile.
The messages log result that you got is what is intended from a single instance document. The test report, on the other hand, is a result from running an XBRL testcases file or testcases index, which usually represents a suite of tests, indicating test case variations, what to load, expected results to pass, etc. If running a whole suite (such as the conformance suite for base spec, XDT, or other), then the test report shows each test, the expected and achieved result, whether the test passes or not, etc. But on a single instance, just the log of messages applies.
In the GUI it is possible to switch the label language and also change label format to name and to Standard Label. Is there a way to get all of those formats as separate columns when calling through command-line?
yep, easy addition, will also add same to the new web interface (it’s common code), wondering, which views are you asking for first (e.g., fact list, tree views, etc)?
I am calling the webserver with …facts?media=xhtml&factListCols=Label,contextRef,unitRef,Dec,Value,EntityScheme,EntityIdentifier,Period,Dimensions. Right now to get the different language labels, I would have to change the system language and call it again.
I’ve tested the parameter &labelLang=ja (or any other language) and it seems to work (one language at a time). It is common to the command line parameters, adding it to the documentation.
Thank you. How can I get the labels to so up in a format of(jpfr-t-cte:…) through the command-line?
I believe I got it. Thank you for your help.
Hi, thanks so much for this I am looking to implement a little validation tool, and I have two questions really,
1. I was wondering if it was possible to validate multiple xbrl instances via the command line?
2. Currently it takes between 20 to 30 seconds to load each instance and then a few seconds to validate it. Would it be possible to cache the xsd, linkbases, etc and use this to validate xbrl instances via the command line?
Any help would be brilliant, thanks
Carey
Yes, but in stages of difficulty. It largely depends on your type of instances.
1) If they are like US-SEC or IFRS, with each instance having independent extensions and linkbases, there is much less to be gained and significant work to preserve in memory what might be shared.
2) If the instances all have precisely the identical linkbases and taxonomies, then it is quite easy and a lot to gain in speed.
3) Several others have found that by profiling the validations, we’ve been able to get the speed improved significantly, if the instances can be tested on our end for this, please contact support@arelle.org by private message.
4) There might be a lot more flexibility in sharing using the web service API, because the command line invocations are separate of each other and don’t share any process or memory. (Or a design could consider a list of multiple instances all sharing the same DTS in one command line call.)
error message from bat file:
Usage: arelleCmdLine.exe [options]
arelleCmdLine.exe: error: no such option: –csvFactCols
Any idea?
Some more info. I have tried both the 32 and 64 bit version of the Windows Arielle Command Line. Neither of them take the –csvFactList as an option. Any idea when this will be supported?
Regards,
Charles
In doing the web services API, the file output was enhanced from just csv, to also html, text, xml, and json (based on the output file name extension), so the parameter was renamed from csvFactList to factListCols. The –help option should have had the latest supported by the code, but anyway the web page and bat file needed editing. Thanks for digging into this. The bat files examples and web page have been updated.
Great tool! Works with the DK-GAAP.
I’ve tested some of the arguments. But it doesn’t work when using:
arelleCmdLine.exe -f “P:\dcca20120101\20120101\entryDanishGAAPBalanceSheetAccountFormIncomeStatementByFunctionIncludingManagementsReviewStatisticsAndTax20120101.xsd” -i “c:\temp\npn.xbrl” -v –facts “c:\temp\npn.csv” –factListCols “Label Name contextRef unitRef Dec Prec Lang Value EntityScheme EntityIdentifier Period Dimensions”
The Name colnne is blank! (I assume that “Name” is the name off the Element from the taxonomy.
this was fixed, thanks for the report
Hi, I’m having problems running the following command line
C:\Program Files\Arelle>arelleCmdLine.exe -f “C:\Users\mauricio.ahumada\Desktop\
XBRL 2012-03\Estados_financieros_(XBRL)60806000_201203.xbrl” -v
I just got several errors similar to the following:
[] HTTP Error 404: Not Found
retrieving http://xbrl.ifrs.org/taxonomy/2011-03-25/generic-link.xsd -
[FileNotLoadable] File can not be loaded: http://xbrl.ifrs.org/taxonomy/2011-03-
25/generic-link.xsd – http://xbrl.ifrs.org/taxonomy/2011-03-25/full_ifrs/ifrs_7_
2011-03-25/gre_ifrs_7_2011-03-25.xml 12
[] HTTP Error 404: Not Found
retrieving http://xbrl.ifrs.org/taxonomy/2011-03-25/generic-reference.xsd -
[FileNotLoadable] File can not be loaded: http://xbrl.ifrs.org/taxonomy/2011-03-
25/generic-reference.xsd – http://xbrl.ifrs.org/taxonomy/2011-03-25/full_ifrs/if
rs_7_2011-03-25/gre_ifrs_7_2011-03-25.xml 13
When using GUI mode, loading and validating XBRL works fine, problem arises from whitin the command line mode.
Greetings!
It looks like these files are not on the ifrs website at this moment. If you have them they could be copied into the cache (on Windows, using tools->internet->manage cache to locate the right directory, or handled by the mappings.xml file in the config directory. (For further help please contact support@arelle.org.)
Hey, I seem to get a syntax error when trying to run the command line. Any ideas, or would it be something I’m doing wrong?
mike@mike-VirtualBox:~/arelle/Arelle$ python arelleCmdLine.py
Traceback (most recent call last):
File "arelleCmdLine.py", line 10, in
from arelle import CntlrCmdLine, CntlrComServer
File "/home/mike/arelle/Arelle/arelle/CntlrCmdLine.py", line 14, in
from arelle import (Cntlr, FileSource, ModelDocument, XmlUtil, Version,
File "/home/mike/arelle/Arelle/arelle/Cntlr.py", line 409
print(logEntry, file=sys.stderr)
^
SyntaxError: invalid syntax
Thanks.
This doesn’t seem to be the latest source code, what platform are you running on, what ws on the command line? Please continue this by e-mail to support@arelle.org
Hi there,
i’ve just tested the –csvPre option and it works just fine. It prints out the standard label. What i would need is the concept name and the namespace to get the exact concept. Would that be possible?
Thanks for your help in advance
for –csvPre, there’s an option to specify custom label roles, and one of those options provides prefixed name instead of label (but not the full namespace). Please try –labelRole XBRL-concept-name. Otherwise we could code a feature to have customizable columns (like the fact list –factListCols) and include a full namespace and name as well.
Hi. First of all, great work, your tool is excellent.
Im having some trouble getting a validation report when calling Arelle through command line. I dont seem able to find a way to write a report showing the errors the XBRL instance has.
I execute this
C:\Program Files\Arelle>arelleCmdLine.exe -f c:\DBNeT\xbrl\out\XBRLMALOS\993010002_201303_I.xbrl -v
The program finishes and exit with to message, output or anything. I know for certaint the instance has some errors, in fact the graphical version of Arelle detects correctly such inconsistencies.
Any help?
Thanks in advance.
Ignore what i just said. Problem was Arelle needs you to explicity tell it how you want the calculations to be done, ie you need the –calcDecimals or –calcDecimals