diff --git a/DOCTOOLS.README.md b/DOCTOOLS.README.md index 5358545e7..8916aacf0 100644 --- a/DOCTOOLS.README.md +++ b/DOCTOOLS.README.md @@ -1,10 +1,10 @@ # Documentation tools cheatsheet for beginners -CFEngine technical documentation is written using a Markdown formatting (see [Markdown Cheatsheet][Markdown Cheatsheet]) and published using a combination of Jekyll and various plugins (see [CFEngine Documentation][CFEngine Documentation]). +CFEngine technical documentation is written using a Markdown formatting (see [Markdown Cheatsheet][Markdown Cheatsheet]) and published using Hugo and our own pre- and post-processing scripts (see [CFEngine Documentation][CFEngine Documentation]). The various Markdown and other files (e.g. images) that make up the volume of CFEngine documentation are held in a master source code repository. Contributors to the documentation make their own changes in local copies of the documentation Markdown on their own machines and then push those changes to a remote branch. -From there a request is made to merge the modifications back into the mainline, from which the Jekyll tooling will be triggered to import and then render the final documentation in HTML. +From there a request is made to merge the modifications back into the mainline, from which the Hugo tooling will be triggered to import and then render the final documentation in HTML. ## Git and GitHub diff --git a/cheatsheet.markdown b/cheatsheet.markdown index e6b987a29..e72b0a762 100644 --- a/cheatsheet.markdown +++ b/cheatsheet.markdown @@ -376,7 +376,7 @@ SELECT +* always run it through Pygments plus the appropriate lexer + +Most important are the `cf3` lexer, as well as `bash`, `console`, - +`diff`, `shell-session` and `postgresql`. But Jekyll supports + +`diff`, `shell-session` and `postgresql`. But Hugo supports +[many more lexers](http://pygments.org/docs/lexers/) + * avoid custom color schemes and hand-coded HTML @@ -397,7 +397,7 @@ index 92555a2..b49c0bb 100644 +* always run it through Pygments plus the appropriate lexer + +Most important are the `cf3` lexer, as well as `bash`, `console`, -+`diff`, `shell-session` and `postgresql`. But Jekyll supports ++`diff`, `shell-session` and `postgresql`. But Hugo supports +[many more lexers](http://pygments.org/docs/lexers/) + * avoid custom color schemes and hand-coded HTML @@ -582,18 +582,11 @@ updated with each new version. Variables that are defined in the front matter (thats the content between the three dashes at the top) or in -[\_config.yaml](https://github.com/cfengine/documentation/blob/master/generator/_config.yml) +[hugo/config.toml](https://github.com/cfengine/documentation/blob/master/hugo/config.toml) can be used directly within markdown. For example this is the '{{< params "CFE_manuals_version" >}}' version of the -documentation. That variable comes from \_config.yaml. - -Since liquid variables look a lot like mustache variables any time you want to -show the actual variables will need to be inside of raw tags. - -{% raw %} -site.CFE_manuals_version {{ site.CFE_manuals_version }} -{% endraw %} +documentation. That variable comes from hugo/config.toml. ## Testing diff --git a/generator/README.org b/generator/README.org index c024dadeb..556ca2b6e 100644 --- a/generator/README.org +++ b/generator/README.org @@ -15,85 +15,29 @@ formatting examples including usage of our custom extensions. :END: For a given branch it can be useful to define variables that can be expanded -anywhere in the documentation. Variables can be defined in [[./_config.yml][=_config.yml=]] and -then referenced in markdown. Jekyll uses the [[https://github.com/Shopify/liquid/wiki][Liquid templating language]] to -process templates. - -[[https://github.com/cfengine/documentation/commit/1793bf6fe8a632e5bb290290cd6044695728f2a6][This commit]] shows liquid variables in use. +anywhere in the documentation. Variables are defined under =[params]= in +[[../hugo/config.toml][=hugo/config.toml=]] and referenced in markdown with the =params= shortcode, for +example ={{< params "CFE_manuals_version" >}}=. * Building Documentation :PROPERTIES: :ID: 4e51b3c2-0970-438d-9587-6a54369533a6 :END: -The documentation is rendered using [[https://jekyllrb.com/][jekyll]]. Jekyll is a ruby based static site -generator and is the engine used by GitHub Pages. +The documentation is rendered using [[https://gohugo.io/][Hugo]]. The Hugo site lives in [[../hugo][=hugo/=]] +and the version used by the build is installed by [[./build/install_hugo.sh][=build/install_hugo.sh=]]. -** Build Environment Setup +** Building Locally :PROPERTIES: :ID: 6b24605a-c9f6-481a-9af5-97aafb68115e :END: -*Note:* The documentation build environment has very particular requirements for -*old* libraries and environment. - -*** Installation -:PROPERTIES: -:ID: 8a488c3a-6870-463d-bb6a-eba032727aba -:END: - -This installation was tested on ubuntu 10.04 with vagrant, ruby 1.9.3 and -Java 7. Make sure your machine has everything according to jekyll requirements: -https://github.com/mojombo/jekyll. - -=git= is required to checkout documentation. - -You also need: -- =java= - for closure compiler -- =python-pygments= 1.5 or later for the code highlighting - -Gems: -- =jekyll-asset-pipeline= -- =closure-compiler= -- =yui-compressor= -- =redcarpet= -- =albino= -- =uglifier= -- =execjs= -- =sanitize= - -If =pandoc= is used: -- =pandoc-ruby= -- =rdiscount= - - -- [[./_setup/start.sh][=_setup/start.sh=]] is a shell script to install these dependencies. It needs to -be reviewed and adjusted to your environment accordingly. Clean setup - -- Review steps described in file [[./_setup/step2.txt][=_setup/step2.txt=]] - -- Create pages folder and checkout [[https://github.com/cfengine/documentation.git][CFEngine documentation]] inside. - -- All *content* files should be inside [[./pages][=pages=]] folder. -- All files must have *=meta=* data on top. See (Page meta data section). -- Templates are saved in the [[./_layouts][=_layouts=]] folder. -- Images and other materials saved in *media* folder. -- JS and css saved in _assets folder. To include new files open - [[./_includes/head.html][=_includes/head.html=]] file. -- Make sure you set correct settings in =_config.yml= for =CFE_OUTPUT= and - =CFE_DIR= - -To build the project change into to the project root folder and run =jekyll=. -Your can find the finished rendered files inside [[./pages][=pages=]] folder - -Run local server and automatically build changes to pages: -Change into the project root folder and run =jekyll --auth --server 4000= -where =4000= is the local port number to run the webserver on. - -*Note:* you must restart server after changes in =_config.yml=. +The easiest way to build the documentation locally is [[../build-locally.sh][=build-locally.sh=]] in the +repository root. It clones the other repositories the build needs into =tmp/= +and runs the same Docker-based pipeline as CI. See the comments at the top of +the script for the available options. -To configure jekyll edit =_config.yml=. See the [[https://github.com/mojombo/jekyll/wiki/Configuration][jekyll Configuration Wiki]] for a -description of all available options. +See [[./build/README.md][=build/README.md=]] for running the container build manually. ** CI Pipelines for Documentation :PROPERTIES: @@ -125,7 +69,6 @@ After all repositories are updated and available the following is executed. #+BEGIN_SRC sh export WRKDIR=`pwd` -$WRKDIR/documentation/generator/_scripts/cfdoc_bootstrap.py master cd $WRKDIR/core NO_CONFIGURE=1 ./autogen.sh #+END_SRC @@ -174,7 +117,7 @@ Once the syntax map has been regenerated [[https://github.com/cfengine/documenta (whatever that means), resolving links, expanding our custom macros and creating printable (dead trees) versions of the documentation. -After pre-processing is complete [[https://github.com/cfengine/documentation/blob/master/generator/_scripts/_run.sh][_run.sh]] performs the =jekyll= build, +After pre-processing is complete [[https://github.com/cfengine/documentation/blob/master/generator/_scripts/_run.sh][_run.sh]] performs the Hugo build and post processes the build with [[https://github.com/cfengine/documentation/blob/master/generator/_scripts/cfdoc_postprocess.py][cfdoc_postprocess.py]] *** Publishing diff --git a/generator/_config.yml b/generator/_config.yml deleted file mode 100644 index 95f2c97b3..000000000 --- a/generator/_config.yml +++ /dev/null @@ -1,48 +0,0 @@ -#auto: true -safe: false -plugins: ./_plugins/ -# baseurl: /docs/master/ -git-repository: github.com/cfengine/documentation -git-branch: "3.27" -#destination: -#layouts: ./_layouts/ -#permalink: /:title.html - -#CFEngine addon -# CFE_OUTPUT turn on special rules for output, see step2.txt about jekyll core hack (https://github.com/cfengine/documentation/blob/master/generator/_setup/step2.txt) -CFE_OUTPUT: true - -#CFE_DIR - output folder if CFE_OUTPUT turned on -CFE_DIR: _site -#to create correct links in sitemap -CFE_manuals_version: "3.27" -cfengine: - branch: "3.27" - core_branch: "3.27.x" - latest_patch_release: 2 - latest_package_build: 1 - vagrant_package_build: 1 - masterfiles_branch: "3.27.x" - -Template: - parse: - line_numbers: true - -vagrant: - version: "2.4.9" - -virtualbox: - version: "7.0.16" - -paginate: 500 - -pygments: true -markdown: redcarpet2 -redcarpet: - extensions: ["no_intra_emphasis", "fenced_code_blocks", "autolink", "strikethrough", "superscript", "tables"] -asset_pipeline: - bundle: true # Default = true - compress: true # Default = true - output_path: assets # Default = assets - display_path: docs/3.27/assets # Default = nil - gzip: false # Default = false diff --git a/generator/_scripts/_create_pdf.sh b/generator/_scripts/_create_pdf.sh deleted file mode 100755 index b3ea9abe4..000000000 --- a/generator/_scripts/_create_pdf.sh +++ /dev/null @@ -1,65 +0,0 @@ -#!/bin/sh -for p in $(echo "$PATH" | tr ":" " ") ;do - if [ -e "$p/weasyprint" ]; then - echo "Found weasyprint, PDF generation started" - else - echo "weasyprint not found in \$PATH. Skipping pdf generation" - exit 0 - fi -done -# -# This script will generate PDF from html files and write them to the _site/pdf -# -# Require: -# - weasyprint installed (see _setup/start.sh) -# - local Apache2 (or any other http server) with configured virtual host -# - LOCAL_WEBSERVER_URL - url for html files -# - LOCAL_WEBROOT - path to the local webroot -# - -if [ -z "$WRKDIR" ] -then - echo Environment WRKDIR is not set, setting it to current working directory - WRKDIR=`pwd` - export WRKDIR -fi - -#html source -DIR=$WRKDIR/documentation/generator/_site - -#output pdf folder -DIR2=$WRKDIR/documentation/generator/_site/pdf - -#git branch -CUR_BRANCH='master' - -LOCAL_WEBROOT=/var/www/docs/$CUR_BRANCH - -LOCAL_WEBSERVER_URL='http://localhost/docs' - -if [ "$LOCAL_WEBROOT" ]; then - sudo mkdir -p $LOCAL_WEBROOT || true -fi - -# copy _site to the local webroot -sudo rm -R $LOCAL_WEBROOT/* -sudo cp -R $DIR/* $LOCAL_WEBROOT/ - - -#get all files and remove / and .html from filename -FILENAME=`find $DIR -type f -name *.html | awk -F $DIR '{print $2}' | cut -d "/" -f 2 | cut -d . -f 1` - -if [ "$DIR2" ]; then - mkdir -p $DIR2 || true -fi - -# do not create PDF for tags or TODO printable-reference -for i in $FILENAME; do - if [ $i != "tags" ] && [ $i != "printable-reference" ] - then - echo '\n' $i '\n' - weasyprint $LOCAL_WEBSERVER_URL/$CUR_BRANCH/$i.html $DIR2/$i.pdf || true - fi -done - -echo "PDF generation finished" diff --git a/generator/_scripts/bootstrap-build.sh b/generator/_scripts/bootstrap-build.sh deleted file mode 100755 index 1e2c71436..000000000 --- a/generator/_scripts/bootstrap-build.sh +++ /dev/null @@ -1,45 +0,0 @@ -#/usr/bin/env bash - -usage() -{ - echo "Usage: $0 [WORKDIR] [BRANCH] - - WORKDIR :: Defaults to current working directory - BRANCH :: Branch to build (defaults to currently checked out branch in current working directory)" - -} - -case $# in - "0" ) - echo "No arguments supplied" - export WRKDIR="$(pwd)" - BRANCH="$(git symbolic-ref --quiet --short HEAD)" - ;; - "1" ) - export WRKDIR=$1 - BRANCH="$(git symbolic-ref --short HEAD 2>/dev/null)" - ;; - "2" ) - export WRKDIR=$1 - BRANCH="$2" - ;; - * ) - echo "Error: Too many arguments" - usage - exit;; -esac - -echo "Set WRKDIR=$WRKDIR" -echo "Set BRANCH=$BRANCH" - -# Make Prepare masterfiles for policy docs -cd $WRKDIR/masterfiles -NO_CONFIGURE=1 ./autogen.sh - -cd $WRKDIR - -$WRKDIR/documentation/generator/_scripts/cfdoc_bootstrap.py $BRANCH - -# Prepare core for syntax docs -cd $WRKDIR/core -NO_CONFIGURE=1 ./autogen.sh diff --git a/generator/_scripts/cfdoc_bootstrap.py b/generator/_scripts/cfdoc_bootstrap.py deleted file mode 100755 index 6ce86a8f3..000000000 --- a/generator/_scripts/cfdoc_bootstrap.py +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/python3 - -# The MIT License (MIT) -# -# Copyright (c) 2013 CFEngine AS -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -# THE SOFTWARE. - -import cfdoc_environment as environment -import cfdoc_git as git -import sys - -config = environment.validate(sys.argv[1]) - -try: - git.createData(config) -except: - print("cfdoc_preprocess: Fatal error generating git tags") - sys.stdout.write(" Exception: ") - print(sys.exc_info()) - exit(1) - -exit(0) diff --git a/generator/_scripts/cfdoc_environment.py b/generator/_scripts/cfdoc_environment.py index ac6e644fb..e7f67853a 100644 --- a/generator/_scripts/cfdoc_environment.py +++ b/generator/_scripts/cfdoc_environment.py @@ -63,23 +63,7 @@ def validate(branch): config["include_directories"].append(config["WORKDIR"] + "/core/tests") config["reference_path"] = config["project_directory"] + "/_references.md" - config["config_path"] = config["project_directory"] + "/_config.yml" - - with open(config["config_path"], "r") as config_file: - lines = config_file.readlines() - for line in lines: - comment = line.find("#") - if comment != -1: - line = line[:comment] - assign = line.split(":") - if assign.__len__() != 2: - continue - if assign[1] == "" or assign[1] == "\n": - continue - key = assign[0].lstrip().rstrip() - value = assign[1].lstrip().rstrip() - - config[key] = value + config["CFE_DIR"] = "_site" print("cfdoc_environment: cwd = " + os.getcwd()) print(" config = ") diff --git a/generator/_scripts/cfdoc_git.py b/generator/_scripts/cfdoc_git.py deleted file mode 100644 index 9f2875000..000000000 --- a/generator/_scripts/cfdoc_git.py +++ /dev/null @@ -1,72 +0,0 @@ -# The MIT License (MIT) -# -# Copyright (c) 2013 CFEngine AS -# -# Permission is hereby granted, free of charge, to any person obtaining a copy -# of this software and associated documentation files (the "Software"), to deal -# in the Software without restriction, including without limitation the rights -# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell -# copies of the Software, and to permit persons to whom the Software is -# furnished to do so, subject to the following conditions: -# -# The above copyright notice and this permission notice shall be included in -# all copies or substantial portions of the Software. -# -# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR -# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, -# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE -# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER -# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, -# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN -# THE SOFTWARE. - -import os - - -def createData(config): - configpath = config["config_path"] - if not os.path.exists(configpath): - print('cfdoc_git: "_config.yml" not found in ' + configpath) - return - - cwd = os.getcwd() - os.chdir(config["markdown_directory"]) - config["branch"] = "master" - try: - git = os.popen("git rev-list -1 HEAD") - while True: - line = git.readline().rstrip() - if line == "": - break - config["revision"] = line - git.close() - except: - print("cfdoc_git: Exception when reading revision") - print("cfdoc_git: cwd = " + os.getcwd()) - - branch = "master" - try: - git = os.popen("git branch --no-color") - while True: - line = git.readline().rstrip() - if line == "": - break - if line.find("*") == 0 and line.find("(") == -1: - branch = line.split(" ")[1].rstrip() - except: - print("cfdoc_git: Exception when reading current branch") - - config["branch"] = branch - print("cfdoc_git: Updating " + configpath) - print(" branch = '" + config["branch"] + "'") - print(" revision = '" + config.get("revision", "NOT FOUND!") + "'") - try: - config_file = open(configpath, "a") - config_file.write('git-branch: "' + config.get("branch", "master") + '"\n') - if "revision" in config: - config_file.write('git-revision: "' + config["revision"] + '"\n') - config_file.close() - except: - print("cfdoc_git: Exception when updating " + configpath) - - os.chdir(cwd) diff --git a/generator/_scripts/cfdoc_preprocess.py b/generator/_scripts/cfdoc_preprocess.py index 20820c546..c5b9b528b 100755 --- a/generator/_scripts/cfdoc_preprocess.py +++ b/generator/_scripts/cfdoc_preprocess.py @@ -26,7 +26,6 @@ import cfdoc_metadata as metadata import cfdoc_linkresolver as linkresolver import cfdoc_macros as macros -import cfdoc_git as git import cfdoc_qa as qa import cfdoc_patch_header_nav as patch_header_nav import cfdoc_references_resolver as references_resolver diff --git a/generator/_scripts/provisioning-install-build-tool-chain.sh b/generator/_scripts/provisioning-install-build-tool-chain.sh deleted file mode 100755 index ff9b89a7d..000000000 --- a/generator/_scripts/provisioning-install-build-tool-chain.sh +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env bash - -# This configuration is a copy from jenkins cloud init in https://ci.cfengine.com/configure - -# Modifications - #/usr/bin/env bash - # usage - -usage() -{ - echo "Usage: $0 [HOME] - - HOME :: Directory where gems are installed exists" - -} - -case $# in - "0" ) - echo "No arguments supplied" - HOME="$HOME" - ;; - "1" ) - HOME="$1" - ;; - * ) - echo "Error: Too many arguments" - usage - exit;; -esac - -set -x - -# All of these commands are executed by the jenkins user. - -sudo apt-get update - -# Docslave specifics -# These packages are needed as a dependency of the nokogiri ruby gem (which in turn is a dependency of the sanitize ruby gem). - -sudo apt-get install -y libxslt-dev libxml2-dev - -# Python is needed for our pre and post processing scripts. - -sudo apt-get install -y python - -# These packages are needed to satisfy rvm requirements. - -sudo apt-get install -y libyaml-dev libsqlite3-dev sqlite3 autoconf libgmp-dev libgdbm-dev libncurses5-dev automake libtool bison pkg-config libffi-dev - -# yui-compressor (ruby gem dependency) is written in java, so we need to install it. - -sudo apt-get install -y default-jdk - -sudo apt-get install -y libssl-dev zlib1g-dev - -# For making cfenigne -sudo apt-get -qy install bison flex binutils build-essential fakeroot ntp dpkg-dev libpam0g-dev liblmdb-dev libpcre3-dev - diff --git a/generator/_scripts/starter_pack-build-docs.sh b/generator/_scripts/starter_pack-build-docs.sh deleted file mode 100644 index 6f4c29cc3..000000000 --- a/generator/_scripts/starter_pack-build-docs.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash - -# build instructions pulled from Jenkins - -export WRKDIR=/northern.tech/cfengine - -cd $WRKDIR/masterfiles -NO_CONFIGURE=1 ./autogen.sh - -cd $WRKDIR - -$WRKDIR/documentation/generator/_scripts/cfdoc_bootstrap.py master - -if dpkg --get-selections | grep -q "^cfengine-nova-hub[[:space:]]*install$" >/dev/null; then - echo Found cfengine-nova-hub package installed, skipping build -else - echo Did not find package cfengine-nova-hub installed, trying to build from source - # Prepare core for syntax docs - cd /northern.tech/cfengine/core - ./configure - make -fi - -export WRKDIR=/northern.tech/cfengine -cd /northern.tech/cfengine/documentation/generator - -# Generate syntax data -source ~/.profile -source ~/.rvm/scripts/rvm - -bash -x ./_regenerate_json.sh - -# Preprocess Documentation with custom macros -./_scripts/cfdoc_preprocess.py master - -# Build docs -bash -x ./_scripts/_run.sh master - -#strip leading slash from assets -#in the _site directory -find ./_site/ -name "*.html" | xargs sed -i "s|/docs/master/||g" diff --git a/generator/_setup/start.sh b/generator/_setup/start.sh deleted file mode 100755 index 64a17568f..000000000 --- a/generator/_setup/start.sh +++ /dev/null @@ -1,51 +0,0 @@ -#!/bin/bash - -## RVM install for better ruby support https://rvm.io/rvm/install/ -# install curl -sudo apt-get install curl - -#download rvm -curl -#L https://get.rvm.io | bash -s stable --autolibs=3 --ruby - -#add rvm to known scripts -source /home/vagrant/.rvm/scripts/rvm - -#install ruby -rvm install ruby-1.9.3 - -#switch to it and make default -rvm --default use 1.9.3 - - -#install build essentials for make -sudo apt-get install build-essential - - -## Install Java used for google closure -sudo apt-get install python-software-properties -sudo add-apt-repository ppa:webupd8team/java -sudo apt-get update -sudo apt-get install oracle-java7-installer - -#install weasyprint for PDF generation (http://weasyprint.org/) - -sudo apt-get install python-dev python-pip python-lxml libgtk2.0-0 libffi-dev --fix-missing -sudo apt-get install python-virtualenv - -sudo pip install --upgrade setuptools - -cd /tmp -curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py - -sudo python get-pip.py -sudo pip install WeasyPrint - -#test -weasyprint http://www.w3.org/TR/CSS21/intro.html CSS21-intro.pdf -s http://weasyprint.org/samples/CSS21-print.css - - -#apache2 -sudo apt-get update -sudo apt-get install apache2 - -#Configure webroot if needed diff --git a/generator/_setup/step2.txt b/generator/_setup/step2.txt deleted file mode 100644 index 7fab7f3f1..000000000 --- a/generator/_setup/step2.txt +++ /dev/null @@ -1,27 +0,0 @@ - -# Add pygment cfengine lexer, see http://zzamboni.org/blog/cfengine3-lexer-for-pygments/ -# in my case file was at /usr/lib/pymodules/python2.6/pygments/lexers - - -## you MUST path core -open folder where you installed jekyll, in my case: ~/.rvm/gems/ruby-1.9.3-head/gems/jekyll-0.12.1/lib/jekyll and find page.rb - -edit code inside write function - def write(dest) - -before line FileUtils.mkdir_p(File.dirname(path)) -add the following: - # Dmitry addon - if site.config['CFE_OUTPUT'] == true && site.config['CFE_DIR'] !='' - dest = File.join(@base, site.config['CFE_DIR']) - if (self.data['alias'] !=nil) - path = File.join(dest, CGI.unescape(self.data['alias'])) - else - path = File.join(dest, CGI.unescape(self.url)) - end - path = File.join(path, "index.html") if self.data['alias'] =~ /\/$/ - - else - # use standard jekyll - path = destination(dest) - end - diff --git a/generator/build/install.sh b/generator/build/install.sh index 846ff19f5..cc743d8b0 100644 --- a/generator/build/install.sh +++ b/generator/build/install.sh @@ -9,24 +9,10 @@ sudo apt-get install -y -qq curl gnupg2 wget git # some of packages below depend on tzdata. Installing it without DEBIAN_FRONTEND=noninteractive causes it to ask where in the world are you, interrupting the build process DEBIAN_FRONTEND=noninteractive sudo --preserve-env=DEBIAN_FRONTEND apt-get install -y -qq tzdata -# Docslave specifics -# These packages are needed as a dependency of the nokogiri ruby gem (which in turn is a dependency of the sanitize ruby gem). - -sudo apt-get install -y -qq libxslt-dev libxml2-dev - # Python is needed for our pre and post processing scripts. sudo apt-get install -y -qq python3 -# These packages are needed to satisfy rvm requirements. - -# hint: to figure out rvm requirements, comment this line and watch output of `rvm install ... ruby`. -# To make script stop at that command, change --autolibs argument to read-fail. -# Also you likely will want to exclude 'libssl1.0-dev' since it's and openssl 1.0 library, which is not shipped on modern distros. -sudo apt-get install -y -qq gawk g++ gcc autoconf automake bison libc6-dev libffi-dev libgdbm-dev libncurses5-dev libsqlite3-dev libtool libyaml-dev make pkg-config sqlite3 zlib1g-dev libgmp-dev libreadline-dev - -sudo apt-get install -y -qq default-jdk - curl -sL https://deb.nodesource.com/setup_21.x -o nodesource_setup.sh sudo bash nodesource_setup.sh sudo apt-get install -y -qq nodejs diff --git a/generator/build/main.sh b/generator/build/main.sh index f4177c5d9..c0418b078 100644 --- a/generator/build/main.sh +++ b/generator/build/main.sh @@ -118,9 +118,6 @@ rm cfengine-nova-hub.deb sudo cp -a /var/cfengine/share/NovaBase/masterfiles "$WRKDIR" sudo chmod -R a+rX "$WRKDIR"/masterfiles -# write current branch into the config.yml -echo "branch: $BRANCH" >> "$WRKDIR"/documentation/generator/_config.yml - # replace %branch% placeholder with actual branch name CONFIG_TOML="$WRKDIR/documentation/hugo/config.toml" if [ "$BRANCH" = "lts" ]; then @@ -144,8 +141,6 @@ fi # Preprocess Documentation with custom macros ./_scripts/cfdoc_preprocess.py "$BRANCH" "$LTS_VERSION" || exit 5 -# rvm commands are insane scripts which pollut output -# so instead of set -x we just echo each command ourselves set +x export LC_ALL=C.UTF-8 diff --git a/generator/build/search/.gitignore b/generator/build/search/.gitignore deleted file mode 100644 index 2ccbe4656..000000000 --- a/generator/build/search/.gitignore +++ /dev/null @@ -1 +0,0 @@ -/node_modules/ diff --git a/generator/build/search/createIndex.js b/generator/build/search/createIndex.js deleted file mode 100644 index e3e7263c2..000000000 --- a/generator/build/search/createIndex.js +++ /dev/null @@ -1,75 +0,0 @@ -const FlexSearch = require("flexsearch"); -const fs = require('fs'); -const {readdir} = require('fs').promises; - -const htmlFilesDir = `${__dirname}/../../_site`; - -String.prototype.stripHtmlTags = function () { - return this.replace(/<\/?[^>]+(>|$)/g, " "); -} - -const getHtmlFiles = async (dir) => - await Promise.all( - (await readdir(dir, {withFileTypes: true})) - // filter out non-html files and tags documents - .filter(file => file.name.includes('.html') && !file.name.includes('tags-')) - .map(file => file.name) - ); - -(async () => { - let index = new FlexSearch.Document({ - document: { - id: 'number', - index: ['content', 'headers', 'title'] - }, - charset: "latin", - tokenize: "full", - matcher: "simple", - cache: true - }) - - const htmlFiles = await getHtmlFiles(htmlFilesDir); - for (const key in htmlFiles) { - const htmlContent = fs.readFileSync(`${htmlFilesDir}/${htmlFiles[key]}`).toString(); - const htmlContentMatch = htmlContent.match(/
([\s\S]*?)
/gm); - const titleMatch = htmlContent.match(/

([\s\S]*?)<\/h1>/gm); - const headersMatch = htmlContent.match(/(.*?)<\/h[1-6]>/gm); - - if (htmlContentMatch == null || titleMatch == null) continue; - const title = titleMatch[0].stripHtmlTags().trim(); - let document = { - number: key, // use sequential number as key to reduce search index size - content: htmlContentMatch[0] - .stripHtmlTags() - .replace(/\s\s+/g, ' ') - .replace(/\n/g, " ") - .replace('Suggest changes', ''), - title, - headers: title, - headersMap: {'top': title} - }; - - if (headersMatch) { - headersMatch.map((header) => { - const idMatch = header.match(/id="(.*?)"/i); - document.headersMap[idMatch[1]] = header.stripHtmlTags(); - document.headers += ` ${header.stripHtmlTags()}`; - }); - } - - let breadCrumbs = ''; - const breadCrumbsMatch = htmlContent.match(/