Différences

Ci-dessous, les différences entre deux révisions de la page.

--- dse:netvibes_cloud_connector [2025/03/11 13:39] – nicoanto
+++ dse:netvibes_cloud_connector [2025/09/02 12:26] (Version actuelle) – [Pré-Requis] nicoanto
@@ Ligne 1: / Ligne 1: @@
 ====== DSE / NETVIBES Cloud Connector ======
-==== Checking initial crawl errors ====
+===== 'er-crawler-manager-cli' script Options =====
+<code>
+Usage: com.ds.apollo.er.crawler.manager.cli.ManagerCliMain [options] [command]
+      [command options]
+  Options:
+    --help
+  Commands:
+    init      Initialize managed crawl operations to the cloud tenant
+      Usage: init
+    destroy      Stop managed crawl operations to the cloud tenant and delete
+            the existing index unit
+      Usage: destroy
+    recrawl      Launch a managed re-crawl to the cloud tenant
+      Usage: recrawl
+    config      List configurations
+      Usage: config [options]
+        Options:
+          --list-crawlable-datasources
+            List crawlable data sources configurations
+            Default: false
+          --list-crawled-datasources
+            List crawled data sources configurations
+            Default: false
+    crawl      Start an unmanaged crawl to the cloud tenant
+      Usage: crawl [options]
+        Options:
+        * --strategy
+            Crawl strategy
+          --monitored
+            True if the crawl should be monitored (true by default)
+            Default: true
+    test-cloud-connection      Test connection to the cloud tenant services
+      Usage: test-cloud-connection
+ </code>
+===== Checking initial crawl errors =====
 Initial crawl job generates a dataset to disk in $CLOUD_CONNECTOR_INSTALL_DIR/$ARCH/er-crawler/storage/datasets/default, containing all successfully crawled document identifiers as well as all errors that happened during that crawl. Objective of this dataset, is to be able to analyze & replay the crawl to recover errors that happened.
 To generate an HTML view of this dataset, you can use the provided command line interface: er-crawler-dataset-cli.
+  * Linux
+<code>$CLOUD_CONNECTOR_INSTALL_DIR/$ARCH/code/command/er-crawler-dataset-cli.sh -D $CLOUD_CONNECTOR_INSTALL_DIR/$ARCH/er-crawler/storage/datasets/default generate-html -o $YOUR_OUTPUT_DIR</code>
+  * Windows
+<code>%CLOUD_CONNECTOR_INSTALL_DIR%\%ARCH%\code\command\er-crawler-dataset-cli.bat -D %CLOUD_CONNECTOR_INSTALL_DIR%\%ARCH%\er-crawler\storage\datasets\default generate-html -o %YOUR_OUTPUT_DIR%</code>
@@ Ligne 12: / Ligne 61: @@
   * [[https://help.3ds.com/2025x/english/DSDoc/EXADfsUserMap/exadfs-m-indexing-onpremise-data-sb.htm?contextscope=cloud&id=c858c5ede1374cc89e191130ac3b3db2|DS Doc - Indexing 3DSpace On-premises Data on the Cloud]]
   * [[https://gitlab.paris.exalead.com/apollo/er-crawler/-/wikis/Cloud-Supplemental|Wiki R&D - Cloud Supplemental]]
+  * [[https://gitlab.paris.exalead.com/apollo/er-crawler/-/wikis/Processing-Configuration|Wiki R&D - Cloud Supplemental - Filtering data]]
   * [[https://dsext001-eu1-215dsi0708-3dswym.3dexperience.3ds.com/community/swym:215dsi0708:DSEXT001:community:-UWQVGuiSeuoZ33WWbTLWQ?content=swym:215dsi0708:DSEXT001:wikitree:34OA6fJ5QwOxg9SBzA9l0Q|EDF Road to Cloud - NETVIBES OnPremise Cloud Connector]]
   * [[https://dsext001-eu1-215dsi0708-3dswym.3dexperience.3ds.com/community/swym:215dsi0708:DSEXT001:community:-UWQVGuiSeuoZ33WWbTLWQ?content=swym:215dsi0708:DSEXT001:wikitree:IHOjbqGPRKKwDT1N52GpeA|EDF Road to Cloud - Administration OP - RAZ & Indexation Full Cloud Connector]]
   * [[https://dsext001-eu1-215dsi0708-3dswym.3dexperience.3ds.com/community/swym:215dsi0708:DSEXT001:community:-UWQVGuiSeuoZ33WWbTLWQ?content=swym:215dsi0708:DSEXT001:wikitree:TEVNkZegQTeaF_PMSN9Xjw|EDF Road to Cloud - (Internal) Unboxing DATA SCIENCE R2023X/R2024x]]
+===== Installation on vdemopro =====
+==== Pré-Requis ====
+En tant que l'utilisateur 'e4apop', dans le répertoire /home/data/RTV/DEMOR424/, il faut :
+  * créer le répertoire '/home/data/RTV/DEMOR424/netvibesCloudConnector'
+  * copier le tar du connecteur et le décompresser dans le répertoire '/home/data/RTV/DEMOR424/netvibesCloudConnector'
+  * y copier le tar de tomee et le décompresser dans le répertoire '/home/data/RTV/DEMOR424/netvibesCloudConnector'
+==== Exécution du script d'installation ====
+Il faut lancer le script : ./NETVIBES_CloudConnector.Linux64/1/StartTUI.sh
+<code>
+[e4apop@vdemopro083abv netvibesCloudConnector]$ ./NETVIBES_CloudConnector.Linux64/1/StartTUI.sh
+CurrentMediaDir initial="./NETVIBES_CloudConnector.Linux64/1"
+CurrentMediaDir="/home/data/RTV/DEMOR424/netvibesCloudConnector/./NETVIBES_CloudConnector.Linux64/1"
+Current operating system: "Linux"
+/usr/bin/lsb_release
+DSY_OS_Release="RedHatEnterprise"
+DSY_OS="linux_a64"
+DSY_LIBPATH_VARNAME="LD_LIBRARY_PATH"
+Check Prerequisites for Linux
+RedHatEnterprise 8.4
+Check Prerequisites for Linux OK.
+ INFO: /home/data/RTV/DEMOR424/netvibesCloudConnector/./NETVIBES_CloudConnector.Linux64/1/inst/linux_a64/code/bin/DSYInsAppliTUI
+ INFO: Installer found in current media: "/home/data/RTV/DEMOR424/netvibesCloudConnector/./NETVIBES_CloudConnector.Linux64/1/inst/linux_a64/code/bin/DSYInsAppliTUI".
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Welcome to the 3DEXPERIENCE R2022x HotFix 6 installation program.
+Please refer to the documentation for the Legal Notices.
+Input !q and hit Enter to quit. Just press Enter to continue.
+Enter selection (default: Next):
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Choose the installation directory to update or an empty directory to do a new installation:
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [/usr/DassaultSystemes/R2022x/NETVIBES_CloudConnector]: /home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Apache TomEE+ installation path
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default []: /home/data/RTV/DEMOR424/netvibesCloudConnector/apache-tomee-plus-8.0.15
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+The embedded Apache TomEE+ will be installed for Testing or Demonstration purposes (not for Production).
+Apache TomEE+ connection port
+Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [8080]: 8080
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Choose the server installation directory.
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [/usr/DassaultSystemes/R2022x/3DSpace]: /home/data/RTV/DEMOR424/server
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the 64-bit Java Development Kit (JDK) 11 path.
+The minimum required Java version is Oracle JDK 11.0.8 / AdoptOpenJDK-OpenJ9 JDK 11.0.8 (or its IBM Semeru successor) / Oracle JDK 17.0.3 / IBM Semeru JDK 17.0.3, but it is recommended to install the latest Update of Java 11.
+To download and install the JDK 11, go to https://www.oracle.com/java/technologies/downloads/#java11 or https://developer.ibm.com/languages/java/semeru-runtimes/downloads/.
+It is strongly recommended to use a Java installation path independent from the Java level.
+After the Java installation completes, select '< Back' then 'Next >' again to auto-detect the Java path.
+Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to continue.
+Default []: /home/data/RTV/DEMOR424/jdk
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the username of the crawl user
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [creator]: creator
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the password for the crawl user
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [****]: DemoCenter01
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Do you want to require users to have a valid CAS session ?
+Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to continue.
+[X] No
+Enter selection:
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the URL of the on cloud 3DRegistry
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [https://GEO-registry.3dx-staging.3ds.com]: https://eu1-registry.3dexperience.3ds.com
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the name of the cloud tenant
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default []: r1132101374770
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the on cloud Workspace ID
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [dw-prj-111111-ws]: ffd38927-6958-4bc5-bc85-b7799e71c2dc
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the Agent ID of an authorized CLM agent to call SGI on the cloud
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default []: fa9f240d-9f14-400a-9d33-da5bf053936b
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+Enter the password for CLM agent
+Input !b to get the previous page. Input !c to clear the default value. Input !q and hit Enter to quit. Just press Enter to continue.
+Default [****]: B<n(QU+S+z)"cbM"X[%v3CN}
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+The installation program has enough information to start.
+Input !b to get the previous page. Input !q and hit Enter to quit. Just press Enter to start installation.
+Files will be installed in the following directory:
+  /home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector
+Apache TomEE+ installation path: /home/data/RTV/DEMOR424/netvibesCloudConnector/apache-tomee-plus-8.0.15
+The embedded Apache TomEE+ will be installed for Testing or Demonstration purposes (not for Production).
+  Apache TomEE+ connection port: 8080
+The server installation directory: /home/data/RTV/DEMOR424/server
+Java Development Kit (JDK) path: /home/data/RTV/DEMOR424/jdk
+Crawl username: 'creator'
+CAS session validation: 'no'
+Cloud Registry URL: 'https://eu1-registry.3dexperience.3ds.com'
+Cloud Tenant Name: 'r1132101374770'
+Cloud Workspace ID: 'ffd38927-6958-4bc5-bc85-b7799e71c2dc'
+Cloud CLM Agent ID: 'fa9f240d-9f14-400a-9d33-da5bf053936b'
+Space available: 64.56 GB
+Space required: 41.18 MB
+Enter selection (default: Install):
+Logs are located in "/home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector/InstallData/log/20250902T103230+0200_CODE_linux_a64_NETVIBES_CloudConnector.media"
+Copying files...
+Installation in progress...
+Installing files...
+CODE\linux_a64\EXAApoERCrawlerInsCrawlPwCfg (161B)
+CODE\linux_a64\EXAApoERCrawlerPreq (63B)
+CODE\linux_a64\EXAApoERCrawlerCheckers (5KB)
+CODE\linux_a64\EXAApoERCrawlerCommons (103KB)
+CODE\linux_a64\EXAApoERCrawlerService (1MB)
+CODE\linux_a64\VENSLF4J (787KB)
+CODE\linux_a64\RegistryClient (41KB)
+CODE\linux_a64\iamCASClient (215KB)
+CODE\linux_a64\ENOJavaKernel (2MB)
+CODE\linux_a64\ENOFCSJavaClient (441KB)
+CODE\linux_a64\VENFasterxmlJackson (4MB)
+CODE\linux_a64\VENGoogleGuava (2MB)
+CODE\linux_a64\VENApacheCommonsIO (489KB)
+CODE\linux_a64\VENApacheCommonsCodec (327KB)
+CODE\linux_a64\VENApacheCommonsHttpClient (1MB)
+CODE\linux_a64\VENGoogleProtoBuf (1MB)
+CODE\linux_a64\VENGson (252KB)
+CODE\linux_a64\VENYammerMetrics (83KB)
+CODE\linux_a64\EXASupplementalPreq (63B)
+CODE\linux_a64\EXASupplementalCommons (3MB)
+CODE\linux_a64\EXAApoERCrawlerPreq4OP (14MB)
+CODE\linux_a64\EXAApoERCrawlerDataSetCLI (44KB)
+CODE\linux_a64\EXAApoERCrawlerManager (183KB)
+CODE\linux_a64\EXAApoERCrawlerManagerCLI (76KB)
+CODE\linux_a64\EXAApoERCrawlerFileSyncSvc (92KB)
+CODE\linux_a64\EXAApoERCrawlerManagerWebapp (6KB)
+CODE\linux_a64\EXAApoERCrawlerServiceWebapp (11KB)
+CODE\linux_a64\EXAApoERCrawlerInstaller (88KB)
+CODE\linux_a64\ERConnectorCommon (239KB)
+CODE\linux_a64\ERAgent (1MB)
+CODE\linux_a64\VENJavaEE_API (4MB)
+CODE\linux_a64\VENApacheJohnzon (253KB)
+CODE\linux_a64\VENApacheCommonsLang (688KB)
+CODE\linux_a64\NETECOP_TP.prd (6KB)
+Installation in progress...
+Setting up properties file...
+Customizing file...
+Executing application...
+Generating er-crawler-manager-cli classpath...
+Generating er-crawler-dataset-cli classpath...
+Generating er-crawler-service-webapp.war...
+Generating er-crawler-manager-webapp.war...
+Customizing file...
+Executing application...
+Installing er-crawler...
+______________________________________
+   Success
+______________________________________
+Installed er-crawler, you can now start your application server
+Once your application server is up, you can initialize the crawl of your tenant by running the following command:
+        '/home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector/linux_a64/code/command/er-crawler-manager-cli.sh init'
+Actions:
+ [1] Ok
+______________________________________
+Please choose an action: 1
+Setting up properties file...
+Installation in progress...
+________________________________________________________________________________
+DEXPERIENCE R2022x HotFix 6
+________________________________________________________________________________
+The installation program has finished installing 3DEXPERIENCE R2022x HotFix 6 on your computer, in /home/data/RTV/DEMOR424/netvibesCloudConnector/NETVIBES_CloudConnector.
+Just press Enter to exit installation.
+Installed Components:
+  NETVIBES ER Crawler On Premise TP
+Enter selection (default: Close):
+</code>
+==== Modifications de la config Apache ====
+En tant que 'root', aller dans le répertoire '/usr/local/reverseproxy/' et modifier le fichier 'conf/httpd.conf
+' en ajoutant, avant le tag </VirtualHost> les lignes suivantes :
+<code>
+  # Analytics Applications
+  ProxyPass        /connector http://vdemoproXXXdsy.extranet.3ds.com:8080
+  ProxyPassReverse /connector http://vdemoproXXXdsy.extranet.3ds.com:8080
+</code>
+Redémarrer le proxy avec une des commandes suivantes
+<code>
+bin/apachectl restart
+</code>
+ou
+<code>
+/usr/local/reverseproxy/bin/httpd -k restart
+</code>