ExternalVendorCode/corda
1
0
Fork 0
mirror of https://github.com/corda/corda.git synced 2025-04-16 07:27:17 +00:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

ENT-2534: Seperate database manager into commands (#1527)

Browse Source 
* Seperate database manager into commands

* Documentation update

* Slightly better naming

* Address review comments

* Review comments

* Address review comments
This commit is contained in:
Anthony Keenan 2018-11-05 14:32:51 +00:00 committed by GitHub
parent 241c045bb9
commit b07cd38186
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
8 changed files with 604 additions and 319 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
docs/source/cli-application-shell-extensions.rst
View File

@ -77,8 +77,10 @@ List of existing CLI applications
List of existing Enterprise CLI applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+----------------------------------------------------------------+--------------------------------------------------------------+--------------------------------+
| Description | JAR name | Alias |
+----------------------------------------------------------------+--------------------------------------------------------------+--------------------------------+
| :ref:`Corda Firewall<firewall-coniguration-file>` | ``corda-firewall-<version>.jar`` | ``corda-firewall --<option>`` |
+----------------------------------------------------------------+--------------------------------------------------------------+--------------------------------+
+----------------------------------------------------------------+--------------------------------------------------------------+----------------------------------+
| Description | JAR name | Alias |
+----------------------------------------------------------------+--------------------------------------------------------------+----------------------------------+
| :doc:`Database Manager<database-management>` | ``corda-tools-database-manager-<version>.jar`` | ``database-manager --<option>`` |
+----------------------------------------------------------------+--------------------------------------------------------------+----------------------------------+
| :doc:`Corda Firewall<firewall-configuration-file>` | ``corda-firewall-<version>.jar`` | ``corda-firewall --<option>`` |
+----------------------------------------------------------------+--------------------------------------------------------------+----------------------------------+

149
docs/source/database-management.rst
View File

@ -159,50 +159,133 @@ Database management tool
The database management tool is distributed as a standalone JAR file named ``tools-database-manager-${corda_version}.jar``.
It is intended to be used by Corda Enterprise node administrators.
Currently it has these features:
The following sections document the available subcommands.
1. ``--create-migration-sql-for-cordapp``: Creates migration scripts for each ``MappedSchema`` in a CorDapp. Each
``MappedSchema`` in a CorDapp installed on a Corda Enterprise node requires the creation of a new table in the
node's database. It is bad practice to apply these changesets to a production database automatically. Instead,
migration scripts must be generated for each schema. These can then be inspected before being applied
2. ``--dry-run``: Inspects the actual SQL statements that will be run as part of a migration job
3. ``--execute-migration``: Runs migration scripts on the node's database
4. ``--release-lock``: Forces the release of database locks. Sometimes, when a node or the database management
tool crashes while running migrations, Liquibase will not release the lock. This can happen during some long
database operations, or when an admin kills the process (this cannot happen during normal operation of a node,
only during the migration process - see: <http://www.liquibase.org/documentation/databasechangeloglock_table.html>)
Creating SQL migration scripts for CorDapps
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It has the following command line options:
The ``create-migration-sql-for-cordapp`` subcommand can be used to create migration scripts for each ``MappedSchema`` in
a CorDapp. Each ``MappedSchema`` in a CorDapp installed on a Corda Enterprise node requires the creation of new tables
in the node's database. It is generally considered bad practice to apply changes to a production database automatically.
Instead, migration scripts can be generated for each schema, which can then be inspected before being applied.
.. table::
Usage:
==================================== =======================================================================
Option Description
==================================== =======================================================================
--help Print help message
--mode Either 'NODE' or 'DOORMAN'. By default 'NODE'
--base-directory(*) The node directory
--config-file The name of the config file, by default 'node.conf'
--doorman-jar-path For internal use only
--create-migration-sql-for-cordapp Create migration files for a CorDapp. You can specify the fully qualified
name of the ``MappedSchema`` class. If not specified it will generate the migration
for all schemas that don't have migrations. The output directory is the
base-directory, where a ``migration`` folder is created.
--dry-run Output the database migration to the specified output file. The output directory
is the base-directory. You can specify a file name or 'CONSOLE' if you want to send the output to the console.
--execute-migration This option will run the database migration on the configured database. This is the
only command that will actually write to the database.
--release-lock Releases whatever locks are on the database change log table, in case shutdown failed.
==================================== =======================================================================
.. code-block:: shell
For example::
database-manager create-migration-sql-for-cordapp [-hvV] [--jar]
[--logging-level=<loggingLevel>]
-b=<baseDirectory>
[-f=<configFile>]
[<schemaClass>]
The ``schemaClass`` parameter can be optionally set to create migrations for a particular class, otherwise migration
schemas will be created for all classes found.
Additional options:
* ``--base-directory``, ``-b``: (Required) The node working directory where all the files are kept (default: ``.``).
* ``--config-file``, ``-f``: The path to the config file. Defaults to ``node.conf``.
* ``--jar``: Place generated migration scripts into a jar.
* ``--verbose``, ``--log-to-console``, ``-v``: If set, prints logging to the console as well as to a file.
* ``--logging-level=<loggingLevel>``: Enable logging at this level and higher. Possible values: ERROR, WARN, INFO, DEBUG, TRACE. Default: INFO.
* ``--help``, ``-h``: Show this help message and exit.
* ``--version``, ``-V``: Print version information and exit.
Executing SQL migration scripts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``execute-migration`` subcommand runs migration scripts on the node's database.
Usage:
.. code-block:: shell
database-manager execute-migration [-hvV] [--doorman-jar-path=<doormanJarPath>]
[--logging-level=<loggingLevel>]
[--mode=<mode>] -b=<baseDirectory>
[-f=<configFile>]
* ``--base-directory``, ``-b``: (Required) The node working directory where all the files are kept (default: ``.``).
* ``--config-file``, ``-f``: The path to the config file. Defaults to ``node.conf``.
* ``--mode``: The operating mode. Possible values: NODE, DOORMAN. Default: NODE.
* ``--doorman-jar-path=<doormanJarPath>``: The path to the doorman JAR.
* ``--verbose``, ``--log-to-console``, ``-v``: If set, prints logging to the console as well as to a file.
* ``--logging-level=<loggingLevel>``: Enable logging at this level and higher. Possible values: ERROR, WARN, INFO, DEBUG, TRACE. Default: INFO.
* ``--help``, ``-h``: Show this help message and exit.
* ``--version``, ``-V``: Print version information and exit.
Executing a dry run of the SQL migration scripts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``dry-run`` subcommand can be used to output the database migration to the specified output file or to the console.
The output directory is the one specified by the ``--base-directory`` parameter.
Usage:
.. code-block:: shell
database-manager dry-run [-hvV] [--doorman-jar-path=<doormanJarPath>]
[--logging-level=<loggingLevel>] [--mode=<mode>]
-b=<baseDirectory> [-f=<configFile>] [<outputFile>]
The ``outputFile`` parameter can be optionally specified determine what file to output the generated SQL to, or use
``CONSOLE`` to output to the console.
Additional options:
* ``--base-directory``, ``-b``: (Required) The node working directory where all the files are kept (default: ``.``).
* ``--config-file``, ``-f``: The path to the config file. Defaults to ``node.conf``.
* ``--mode``: The operating mode. Possible values: NODE, DOORMAN. Default: NODE.
* ``--doorman-jar-path=<doormanJarPath>``: The path to the doorman JAR.
* ``--verbose``, ``--log-to-console``, ``-v``: If set, prints logging to the console as well as to a file.
* ``--logging-level=<loggingLevel>``: Enable logging at this level and higher. Possible values: ERROR, WARN, INFO, DEBUG, TRACE. Default: INFO.
* ``--help``, ``-h``: Show this help message and exit.
* ``--version``, ``-V``: Print version information and exit.
Releasing database locks
~~~~~~~~~~~~~~~~~~~~~~~~
The ``release-lock`` subcommand forces the release of database locks. Sometimes, when a node or the database management
tool crashes while running migrations, Liquibase will not release the lock. This can happen during some long
database operations, or when an admin kills the process (this cannot happen during normal operation of a node,
only during the migration process - see: <http://www.liquibase.org/documentation/databasechangeloglock_table.html>)
Usage:
.. code-block:: shell
database-manager release-lock [-hvV] [--doorman-jar-path=<doormanJarPath>]
[--logging-level=<loggingLevel>] [--mode=<mode>]
-b=<baseDirectory> [-f=<configFile>]
Additional options:
* ``--base-directory``, ``-b``: (Required) The node working directory where all the files are kept (default: ``.``).
* ``--config-file``, ``-f``: The path to the config file. Defaults to ``node.conf``.
* ``--mode``: The operating mode. Possible values: NODE, DOORMAN. Default: NODE.
* ``--doorman-jar-path=<doormanJarPath>``: The path to the doorman JAR.
* ``--verbose``, ``--log-to-console``, ``-v``: If set, prints logging to the console as well as to a file.
* ``--logging-level=<loggingLevel>``: Enable logging at this level and higher. Possible values: ERROR, WARN, INFO, DEBUG, TRACE. Default: INFO.
* ``--help``, ``-h``: Show this help message and exit.
* ``--version``, ``-V``: Print version information and exit.
Database Manager shell extensions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``install-shell-extensions`` subcommand can be used to install the ``database-manager`` alias and auto completion for
bash and zsh. See :doc:`cli-application-shell-extensions` for more info.
java -jar tools-database-manager-3.0.0.jar --base-directory /path/to/node --execute-migration
.. note:: When running the database management tool, prefer using absolute paths when specifying the "base-directory".
.. warning:: It is good practice for node operators to backup the database before upgrading to a new version.
Examples
--------

139
tools/dbmigration/src/main/kotlin/com/r3/corda/dbmigration/CreateMigrationSqlForCordappsCli.kt Normal file
View File

@ -0,0 +1,139 @@
package com.r3.corda.dbmigration
import net.corda.cliutils.CliWrapperBase
import net.corda.cliutils.ExitCodes
import net.corda.core.internal.copyTo
import net.corda.core.internal.div
import net.corda.node.services.persistence.MigrationExporter
import net.corda.nodeapi.internal.MigrationHelpers
import picocli.CommandLine.*
import java.io.File
import java.io.FileOutputStream
import java.nio.file.Path
import java.util.*
import java.util.jar.JarOutputStream
import java.util.zip.ZipEntry
class CreateMigrationSqlForCordappOptions : SharedDbManagerOptions {
override var mode: Mode = Mode.NODE // This can only be run on a node, don't expose as configurable via picocli
override var doormanJarPath: Path? = null
@Option(
names = ["-b", "--base-directory"],
description = ["The output directory, where a `migration` folder is created."],
required = true
)
override var baseDirectory: Path? = null
@Option(
names = ["-f", "--config-file"],
description = ["The name of the config file. By default 'node.conf' for a simple node and 'network-management.conf' for a doorman."]
)
override var configFile: String? = null
@Option(
names = ["--$JAR_OUTPUT"],
description = ["Place generated migration scripts into a jar."]
)
var jarOutput: Boolean = false
}
class CreateMigrationSqlForCordappsCli : CliWrapperBase(CREATE_MIGRATION_CORDAPP, "Create migration files for a CorDapp.") {
@Mixin
var cmdLineOptions = CreateMigrationSqlForCordappOptions()
@Parameters(arity = "0..1", description = ["You can specify the fully qualified name of the `MappedSchema` class. If not specified it will generate the migration for all schemas that don't have migrations."])
var schemaClass: String = ""
private val db by lazy { cmdLineOptions.toConfig() }
override fun runProgram(): Int {
val outputSchemaMigrations = mutableListOf<Pair<Class<*>, Path>>()
if (schemaClass != "") {
generateMigrationFileForSchema(schemaClass)?.let { outputSchemaMigrations.add(it) }
} else {
outputSchemaMigrations.addAll(db.schemas.asSequence().filter { MigrationHelpers.getMigrationResource(it, db.classLoader) == null }.mapNotNull {
generateMigrationFileForSchema(it.javaClass.name)
}.toList())
}
if (cmdLineOptions.jarOutput) {
val groupedBySourceJar = outputSchemaMigrations.asSequence().map { (klazz, path) ->
//get the source jar for this particular schema
klazz.protectionDomain.codeSource to path
}.filter {
//filter all entries without a code source
it.first != null
}.map {
//convert codesource to a File
File(it.first.location.toURI()) to it.second
}.groupBy {
//group by codesource File
it.first
}.map {
// convert File into a filename for the codesource
// if run from within an IDE, there is possibility of some schemas to be
// loaded from a build folder
// so we must handle this case
val fileName = if (it.key.name.endsWith("jar")) {
it.key.path
} else {
"unknown-cordapp.jar"
}
fileName to it.value.map { it.second }
}.toList().toMap()
groupedBySourceJar.entries.forEach { (_, migrationFiles) ->
migrationFiles.forEach {
//use System.out directly instead of logger as we MUST output to screen due to user input requirement
System.out.println("##### ${it.fileName} #####")
it.copyTo(System.out)
System.out.println("Is this output as expected? [y/N]")
validateUserResponse()
}
}
System.out.println("""There is potential for data corruption.
Please check the scripts are valid before running.
If there are issues, rerun the command without the --jar parameter,
edit the resulting file and manually create the jar.
Are you sure that the migration scripts are acceptable? [y/N]""")
if (!validateUserResponse()) return ExitCodes.FAILURE
groupedBySourceJar.entries.forEach { (jar, migrationFiles) ->
val sourceJarFile = File(jar)
val migrationJarFile = File(sourceJarFile.parent, "migration-${sourceJarFile.name}")
JarOutputStream(FileOutputStream(migrationJarFile)).use { jos ->
migrationLogger.info("Creating migration CorDapp at: ${migrationJarFile.absolutePath}")
migrationFiles.map { ZipEntry("migration/${it.fileName}") }.forEach {
jos.putNextEntry(it)
}
}
}
}
return ExitCodes.SUCCESS
}
private fun generateMigrationFileForSchema(schemaClass: String): Pair<Class<*>, Path>? {
migrationLogger.info("Creating database migration files for schema: $schemaClass into ${(db.baseDirectory / "migration").toString().trim()}")
var liquiBaseOutput: Pair<Class<*>, Path>? = null
try {
db.runWithDataSource {
liquiBaseOutput = MigrationExporter(db.baseDirectory, db.config.dataSourceProperties, db.classLoader, it).generateMigrationForCorDapp(schemaClass)
}
} catch (e: Exception) {
wrappedError("Could not generate migration for $schemaClass: ${e.message}", e)
}
return liquiBaseOutput
}
private fun validateUserResponse(): Boolean {
val userInput = Scanner(System.`in`).nextLine().toLowerCase()
if (userInput != "y" && userInput != "yes") {
migrationLogger.warn("Quitting due to user input")
return false
}
return true
}
}

183
tools/dbmigration/src/main/kotlin/com/r3/corda/dbmigration/DbManagerOptions.kt Normal file
View File

@ -0,0 +1,183 @@
package com.r3.corda.dbmigration
import com.typesafe.config.Config
import com.typesafe.config.ConfigFactory
import com.typesafe.config.ConfigParseOptions
import net.corda.core.internal.div
import net.corda.core.internal.exists
import net.corda.core.schemas.MappedSchema
import net.corda.node.internal.cordapp.JarScanningCordappLoader
import net.corda.node.services.config.configOf
import net.corda.node.services.schema.NodeSchemaService
import picocli.CommandLine.Option
import java.net.URLClassLoader
import java.nio.file.Path
//command line arguments
const val DOORMAN_JAR_PATH = "doorman-jar-path"
const val EXECUTE_MIGRATION = "execute-migration"
const val DRY_RUN = "dry-run"
const val CREATE_MIGRATION_CORDAPP = "create-migration-sql-for-cordapp"
const val JAR_OUTPUT = "jar"
const val RELEASE_LOCK = "release-lock"
enum class Mode {
NODE, DOORMAN
}
fun SharedDbManagerOptions.toConfig(): DbManagerConfiguration {
return if (this.mode == Mode.DOORMAN) {
requireNotNull(this.doormanJarPath) { "If running against the doorman you must provide the --$DOORMAN_JAR_PATH" }
DoormanDbManagerConfiguration(this)
} else {
NodeDbManagerConfiguration(this)
}
}
class DoormanDbManagerConfiguration(cmdLineOptions: SharedDbManagerOptions) : DbManagerConfiguration(cmdLineOptions) {
override val defaultConfigFileName get() = "network-management.conf"
private val doormanFatJarPath by lazy {
val fatJarPath = cmdLineOptions.doormanJarPath!!
if (!fatJarPath.exists()) {
error("Could not find the doorman JAR in location: '$fatJarPath'.")
}
fatJarPath
}
override val parsedConfig: Config by lazy {
ConfigFactory.parseFile(configFile.toFile()).resolve()
}
private fun loadMappedSchema(schemaName: String, classLoader: ClassLoader) = classLoader.loadClass(schemaName).kotlin.objectInstance as MappedSchema
override val schemas: Set<MappedSchema> by lazy {
val doormanSchema = "com.r3.corda.networkmanage.common.persistence.NetworkManagementSchemaServices\$SchemaV1"
setOf(loadMappedSchema(doormanSchema, classLoader))
}
private fun classLoaderFromJar(jarPath: Path): ClassLoader = URLClassLoader(listOf(jarPath.toUri().toURL()).toTypedArray())
override val classLoader by lazy { classLoaderFromJar(doormanFatJarPath) }
}
class NodeDbManagerConfiguration(cmdLineOptions: SharedDbManagerOptions) : DbManagerConfiguration(cmdLineOptions) {
private val cordappsFolder by lazy { baseDirectory / "cordapps" }
private val cordappSchemas by lazy { cordappLoader.cordappSchemas }
private val cordappLoader by lazy { JarScanningCordappLoader.fromDirectories(setOf(baseDirectory, cordappsFolder)) }
override val defaultConfigFileName get() = "node.conf"
override val classLoader by lazy { cordappLoader.appClassLoader }
override val schemas: Set<MappedSchema> by lazy { NodeSchemaService(extraSchemas = cordappSchemas).schemaOptions.keys }
override val parsedConfig: Config by lazy {
ConfigFactory.parseFile(configFile.toFile())
.withFallback(configOf("baseDirectory" to cmdLineOptions.baseDirectory.toString()))
.withFallback(ConfigFactory.parseResources("reference.conf", ConfigParseOptions.defaults().setAllowMissing(true)))
.resolve()
}
}
interface SharedDbManagerOptions {
var mode: Mode
var baseDirectory: Path?
var configFile: String?
var doormanJarPath: Path?
fun copyFrom(other: LegacyDbManagerOptions) {
this.mode = other.mode
this.baseDirectory = other.baseDirectory
this.configFile = other.configFile
this.doormanJarPath = other.doormanJarPath
}
}
class DbManagerOptions : SharedDbManagerOptions {
@Option(
names = ["--mode"],
description = ["The operating mode. \${COMPLETION-CANDIDATES}"]
)
override var mode: Mode = Mode.NODE
@Option(
names = ["-b", "--base-directory"],
description = ["The node or doorman directory."],
required = true
)
override var baseDirectory: Path? = null
@Option(
names = ["-f", "--config-file"],
description = ["The name of the config file. By default 'node.conf' for a simple node and 'network-management.conf' for a doorman."]
)
override var configFile: String? = null
@Option(
names = ["--$DOORMAN_JAR_PATH"],
description = ["The path to the doorman JAR."]
)
override var doormanJarPath: Path? = null
}
class LegacyDbManagerOptions : SharedDbManagerOptions {
@Option(
names = ["--mode"],
description = ["DEPRECATED. The operating mode. \${COMPLETION-CANDIDATES}"],
hidden = true
)
override var mode: Mode = Mode.NODE
// --base-directory needs to be set as not required in the root command, otherwise picocli wants you to enter it twice,
// once for the base command and once for the subcommand
@Option(
names = ["-b", "--base-directory"],
description = ["DEPRECATED. The node or doorman directory."],
hidden = true
)
override var baseDirectory: Path? = null
@Option(
names = ["-f", "--config-file"],
description = ["DEPRECATED. The name of the config file. By default 'node.conf' for a simple node and 'network-management.conf' for a doorman."],
hidden = true
)
override var configFile: String? = null
@Option(
names = ["--$DOORMAN_JAR_PATH"],
description = ["DEPRECATED. The path to the doorman JAR."],
hidden = true
)
override var doormanJarPath: Path? = null
@Option(
names = ["--$EXECUTE_MIGRATION"],
description = ["DEPRECATED. This option will run the database migration on the configured database. This is the only command that will actually write to the database."],
hidden = true
)
var executeMigration: Boolean = false
@Option(
names = ["--$DRY_RUN"],
arity = "0..1",
description = ["DEPRECATED. Output the database migration to the specified output file.",
"The output directory is the base-directory.",
"You can specify a file name or 'CONSOLE' if you want to send the output to the console."],
hidden = true
)
var dryRun: String? = null
@Option(
names = ["--$CREATE_MIGRATION_CORDAPP"],
arity = "0..1",
description = ["DEPRECATED. Create migration files for a CorDapp.",
"You can specify the fully qualified name of the `MappedSchema` class. If not specified it will generate the migration for all schemas that don't have migrations.",
"The output directory is the base-directory, where a `migration` folder is created."],
hidden = true
)
var createMigrationSqlForCordappPath: String? = null
val createMigrationSqlForCordapp: Boolean get() = createMigrationSqlForCordappPath != null
@Option(
names = ["--$RELEASE_LOCK"],
description = ["DEPRECATED. Releases whatever locks are on the database change log table, in case shutdown failed."],
hidden = true
)
var releaseLock: Boolean = false
}

38
tools/dbmigration/src/main/kotlin/com/r3/corda/dbmigration/DryRunCli.kt Normal file
View File

@ -0,0 +1,38 @@
package com.r3.corda.dbmigration
import net.corda.cliutils.CliWrapperBase
import net.corda.cliutils.ExitCodes
import picocli.CommandLine
import java.io.File
import java.io.FileWriter
import java.io.PrintWriter
import java.io.Writer
import java.nio.file.Path
import java.text.SimpleDateFormat
import java.util.*
class DryRunCli : CliWrapperBase(DRY_RUN, description = "Output the database migration to the specified output file. The output directory is the base-directory.") {
@CommandLine.Mixin
var cmdLineOptions = DbManagerOptions()
@CommandLine.Parameters(arity = "0..1", description = ["You can specify a file name or 'CONSOLE' if you want to send the output to the console."])
var outputFile: String = ""
private fun getMigrationOutput(baseDirectory: Path): Writer {
return when (outputFile) {
"" -> FileWriter(File(baseDirectory.toFile(), "migration${SimpleDateFormat("yyyyMMddHHmmss").format(Date())}.sql"))
CONSOLE -> PrintWriter(System.out)
else -> FileWriter(File(baseDirectory.toFile(), outputFile))
}
}
override fun runProgram(): Int {
val db = cmdLineOptions.toConfig()
val writer = getMigrationOutput(db.baseDirectory)
migrationLogger.info("Exporting the current database migrations ...")
db.runMigrationCommand(db.schemas) { migration, _ ->
migration.generateMigrationScript(writer)
}
return ExitCodes.SUCCESS
}
}

18
tools/dbmigration/src/main/kotlin/com/r3/corda/dbmigration/ExecuteMigrationsCli.kt Normal file
View File

@ -0,0 +1,18 @@
package com.r3.corda.dbmigration
import net.corda.cliutils.CliWrapperBase
import net.corda.cliutils.ExitCodes
import net.corda.node.services.persistence.DBCheckpointStorage
import picocli.CommandLine.Mixin
class ExecuteMigrationsCli : CliWrapperBase(EXECUTE_MIGRATION, "This option will run the database migration on the configured database. This is the only command that will actually write to the database.") {
@Mixin
var cmdLineOptions = DbManagerOptions()
override fun runProgram(): Int {
val db = cmdLineOptions.toConfig()
migrationLogger.info("Running the database migration on ${cmdLineOptions.baseDirectory}")
db.runMigrationCommand(db.schemas) { migration, dataSource -> migration.runMigration(dataSource.connection.use { DBCheckpointStorage().getCheckpointCount(it) != 0L }) }
return ExitCodes.SUCCESS
}
}

365
tools/dbmigration/src/main/kotlin/com/r3/corda/dbmigration/Launcher.kt
View File

@ -2,135 +2,40 @@
package com.r3.corda.dbmigration
import com.typesafe.config.ConfigFactory
import com.typesafe.config.ConfigParseOptions
import net.corda.cliutils.CordaCliWrapper
import net.corda.cliutils.ExitCodes
import net.corda.cliutils.start
import net.corda.core.internal.copyTo
import com.typesafe.config.Config
import net.corda.cliutils.*
import net.corda.core.internal.Emoji
import net.corda.core.internal.div
import net.corda.core.internal.exists
import net.corda.core.schemas.MappedSchema
import net.corda.node.internal.DataSourceFactory.createDatasourceFromDriverJarFolders
import net.corda.node.internal.cordapp.JarScanningCordappLoader
import net.corda.node.services.config.ConfigHelper
import net.corda.node.services.config.NodeConfigurationImpl
import net.corda.node.services.config.configOf
import net.corda.node.services.persistence.DBCheckpointStorage
import net.corda.node.services.persistence.MigrationExporter
import net.corda.node.services.schema.NodeSchemaService
import net.corda.nodeapi.internal.MigrationHelpers
import net.corda.nodeapi.internal.config.UnknownConfigKeysPolicy
import net.corda.nodeapi.internal.config.parseAs
import net.corda.nodeapi.internal.persistence.CheckpointsException
import net.corda.nodeapi.internal.persistence.DatabaseConfig
import net.corda.nodeapi.internal.persistence.SchemaMigration
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import picocli.CommandLine.Mixin
import picocli.CommandLine.Option
import java.io.*
import java.net.URLClassLoader
import java.nio.file.Path
import java.nio.file.Paths
import java.text.SimpleDateFormat
import java.util.*
import java.util.jar.JarOutputStream
import java.util.zip.ZipEntry
import javax.sql.DataSource
//command line arguments
const val DOORMAN_JAR_PATH = "doorman-jar-path"
const val EXECUTE_MIGRATION = "execute-migration"
const val DRY_RUN = "dry-run"
const val CREATE_MIGRATION_CORDAPP = "create-migration-sql-for-cordapp"
const val JAR_OUTPUT = "jar"
const val RELEASE_LOCK = "release-lock"
// output type
const val CONSOLE = "CONSOLE"
// initialise loggers lazily as some configuration is changed on startup and if loggers are already initialised it will be ignored
private val migrationLogger by lazy { LoggerFactory.getLogger("migration.tool") }
private val errorLogger by lazy { LoggerFactory.getLogger("errors") }
private enum class Mode {
NODE, DOORMAN
}
private class DbManagementToolOptions {
@Option(
names = ["--mode"],
description = ["The operating mode. \${COMPLETION-CANDIDATES}"]
)
var mode: Mode = Mode.NODE
@Option(
names = ["-b", "--base-directory"],
description = ["The node or doorman directory."]
)
var baseDirectory: Path? = null
@Option(
names = ["-f", "--config-file"],
description = ["The name of the config file. By default 'node.conf' for a simple node and 'network-management.conf' for a doorman."]
)
var configFile: String? = null
@Option(
names = ["--$DOORMAN_JAR_PATH"],
description = ["The path to the doorman JAR."]
)
var doormanJarPath: Path? = null
@Option(
names = ["--$EXECUTE_MIGRATION"],
description = ["This option will run the database migration on the configured database. This is the only command that will actually write to the database."]
)
var executeMigration: Boolean = false
@Option(
names = ["--$DRY_RUN"],
arity = "0..1",
description = ["Output the database migration to the specified output file.",
"The output directory is the base-directory.",
"You can specify a file name or 'CONSOLE' if you want to send the output to the console."]
)
var dryRun: String? = null
@Option(
names = ["--$CREATE_MIGRATION_CORDAPP"],
arity = "0..1",
description = ["Create migration files for a CorDapp.",
"You can specify the fully qualified name of the `MappedSchema` class. If not specified it will generate the migration for all schemas that don't have migrations.",
"The output directory is the base-directory, where a `migration` folder is created."]
)
var createMigrationSqlForCordappPath: String? = null
@Option(
names = ["--$JAR_OUTPUT"],
arity = "0..0",
description = ["Place generated migration scripts into a jar"]
)
var jarOutput: Boolean = false
val createMigrationSqlForCordapp: Boolean get() = createMigrationSqlForCordappPath != null
@Option(
names = ["--$RELEASE_LOCK"],
description = ["Releases whatever locks are on the database change log table, in case shutdown failed."]
)
var releaseLock: Boolean = false
}
val migrationLogger: Logger by lazy { LoggerFactory.getLogger("migration.tool") }
val errorLogger: Logger by lazy { LoggerFactory.getLogger("errors") }
fun main(args: Array<String>) {
DbManagementTool().start(args)
}
data class Configuration(val dataSourceProperties: Properties, val database: DatabaseConfig, val jarDirs: List<String> = emptyList())
private class DbManagementTool : CordaCliWrapper("database-manager", "The Corda database management tool.") {
@Mixin
var cmdLineOptions = DbManagementToolOptions()
var cmdLineOptions = LegacyDbManagerOptions()
private fun checkOnlyOneCommandSelected() {
val selectedOptions = mutableListOf<String>()
@ -141,187 +46,70 @@ private class DbManagementTool : CordaCliWrapper("database-manager", "The Corda
require(selectedOptions.count() != 0) { "You must call database-manager with a command option. See --help for further info." }
require(selectedOptions.count() == 1) { "You cannot call more than one of: ${selectedOptions.joinToString(", ")}. See --help for further info." }
if (cmdLineOptions.jarOutput){
require(cmdLineOptions.createMigrationSqlForCordapp) { "You must specify --$CREATE_MIGRATION_CORDAPP to use the --$JAR_OUTPUT option" }
}
}
private val dryRunCli by lazy { DryRunCli() }
private val executeMigrationsCli by lazy { ExecuteMigrationsCli() }
private val createMigrationSqlForCordappsCli by lazy { CreateMigrationSqlForCordappsCli() }
private val releaseLockCli by lazy { ReleaseLockCli() }
override fun additionalSubCommands() = setOf(dryRunCli, executeMigrationsCli, createMigrationSqlForCordappsCli, releaseLockCli)
override fun runProgram(): Int {
// The database manager should be invoked using one of the vaild subcommands. If the old --flag based options are used,
// this is the entry point that will be used. It does some additional validation and then delegates to the relevant
// subcommand. It should be removed in a future version.
return runLegacyCommands()
}
fun runLegacyCommands(): Int {
checkOnlyOneCommandSelected()
require(cmdLineOptions.baseDirectory != null) { "You must specify a base directory" }
fun baseDirectory() = cmdLineOptions.baseDirectory?.toAbsolutePath()?.normalize()!!
fun configFile(defaultCfgName: String) = baseDirectory() / (cmdLineOptions.configFile ?: defaultCfgName)
when {
cmdLineOptions.mode == Mode.NODE -> {
val baseDirectory = baseDirectory()
if (!baseDirectory.exists()) {
error("Could not find base-directory: '$baseDirectory'.")
}
val config = configFile("node.conf")
if (!config.exists()) {
error("Not a valid node folder. Could not find the config file: '$config'.")
}
val cordappsFolder = baseDirectory / "cordapps"
val cordappLoader = JarScanningCordappLoader.fromDirectories(setOf(baseDirectory, cordappsFolder))
val schemaService = NodeSchemaService(extraSchemas = cordappLoader.cordappSchemas)
handleCommand(baseDirectory, config, cmdLineOptions.mode, cordappLoader.appClassLoader, schemaService.schemaOptions.keys, cmdLineOptions.jarOutput)
}
cmdLineOptions.mode == Mode.DOORMAN -> {
if (cmdLineOptions.doormanJarPath != null) {
error("The $DOORMAN_JAR_PATH argument is required when running in doorman mode.")
}
val fatJarPath = cmdLineOptions.doormanJarPath!!
if (!fatJarPath.exists()) {
error("Could not find the doorman jar in location: '$fatJarPath'.")
}
val doormanClassloader = classLoaderFromJar(fatJarPath)
val doormanSchema = "com.r3.corda.networkmanage.common.persistence.NetworkManagementSchemaServices\$SchemaV1"
val schema = loadMappedSchema(doormanSchema, doormanClassloader)
handleCommand(baseDirectory(), configFile("network-management.conf"), cmdLineOptions.mode, doormanClassloader, setOf(schema), false)
}
}
migrationLogger.info("Done")
return ExitCodes.SUCCESS
}
private fun handleCommand(baseDirectory: Path, configFile: Path, mode: Mode, classLoader: ClassLoader, schemas: Set<MappedSchema>, jarOutput: Boolean) {
val parsedConfig = ConfigFactory.parseFile(configFile.toFile()).resolve().let {
if (mode == Mode.NODE) {
it.withFallback(configOf("baseDirectory" to baseDirectory.toString()))
.withFallback(ConfigFactory.parseResources("reference.conf", ConfigParseOptions.defaults().setAllowMissing(true)))
.resolve()
} else {
it
}
}
val config = parsedConfig.parseAs(Configuration::class, UnknownConfigKeysPolicy.IGNORE::handle)
fun runMigrationCommand(withMigration: (SchemaMigration, DataSource) -> Unit): Unit = runWithDataSource(config, baseDirectory, classLoader) { dataSource ->
withMigration(SchemaMigration(schemas, dataSource, config.database, classLoader), dataSource)
}
when {
cmdLineOptions.releaseLock -> runWithDataSource(ConfigFactory.parseFile(configFile.toFile()).resolve().parseAs(Configuration::class, UnknownConfigKeysPolicy.IGNORE::handle), baseDirectory, classLoader) {
SchemaMigration(emptySet(), it, config.database, Thread.currentThread().contextClassLoader).forceReleaseMigrationLock()
cmdLineOptions.releaseLock -> {
printWarning("The --$RELEASE_LOCK flag has been deprecated and will be removed in a future version. Use the $RELEASE_LOCK command instead.")
releaseLockCli.cmdLineOptions.copyFrom(cmdLineOptions)
return releaseLockCli.runProgram()
}
cmdLineOptions.dryRun != null -> {
val writer = getMigrationOutput(baseDirectory)
migrationLogger.info("Exporting the current db migrations ...")
runMigrationCommand { migration, _ ->
migration.generateMigrationScript(writer)
}
printWarning("The --$DRY_RUN option has been deprecated and will be removed in a future version. Use the $DRY_RUN command instead.")
dryRunCli.cmdLineOptions.copyFrom(cmdLineOptions)
dryRunCli.outputFile = cmdLineOptions.dryRun!!
return dryRunCli.runProgram()
}
cmdLineOptions.executeMigration -> {
migrationLogger.info("Running the database migration on $baseDirectory")
runMigrationCommand { migration, dataSource -> migration.runMigration(dataSource.connection.use { DBCheckpointStorage().getCheckpointCount(it) != 0L }) }
printWarning("The --$EXECUTE_MIGRATION option has been deprecated and will be removed in a future version. Use the $EXECUTE_MIGRATION command instead.")
executeMigrationsCli.cmdLineOptions.copyFrom(cmdLineOptions)
return executeMigrationsCli.runProgram()
}
cmdLineOptions.createMigrationSqlForCordapp && (mode == Mode.NODE) -> {
// do we really want to creating captured functions here.
fun generateMigrationFileForSchema(schemaClass: String): Pair<Class<*>, Path>? {
migrationLogger.info("Creating database migration files for schema: $schemaClass into ${(baseDirectory / "migration").toString().trim()}")
var liquiBaseOutput: Pair<Class<*>, Path>? = null
try {
runWithDataSource(config, baseDirectory, classLoader) {
liquiBaseOutput = MigrationExporter(baseDirectory, config.dataSourceProperties, classLoader, it).generateMigrationForCorDapp(schemaClass)
}
} catch (e: Exception) {
wrappedError("Could not generate migration for $schemaClass: ${e.message}", e)
}
return liquiBaseOutput
}
val outputSchemaMigrations = mutableListOf<Pair<Class<*>, Path>>()
if (cmdLineOptions.createMigrationSqlForCordappPath != "") {
generateMigrationFileForSchema(cmdLineOptions.createMigrationSqlForCordappPath!!)?.let { outputSchemaMigrations.add(it) }
} else {
outputSchemaMigrations.addAll(schemas.filter { true || MigrationHelpers.getMigrationResource(it, classLoader) == null }.mapNotNull {
generateMigrationFileForSchema(it.javaClass.name)
})
}
if (jarOutput) {
val groupedBySourceJar = outputSchemaMigrations.map { (klazz, path) ->
//get the source jar for this particular schema
klazz.protectionDomain.codeSource to path
}.filter {
//filter all entries without a code source
it.first != null
}.map {
//convert codesource to a File
File(it.first.location.toURI()) to it.second
}.groupBy {
//group by codesource File
it.first
}.map {
// convert File into a filename for the codesource
// if run from within an IDE, there is possibility of some schemas to be
// loaded from a build folder
// so we must handle this case
val fileName = if (it.key.name.endsWith("jar")) {
it.key.path
} else {
"unknown-cordapp.jar"
}
fileName to it.value.map { it.second }
}.toMap()
groupedBySourceJar.entries.forEach { (_, migrationFiles) ->
migrationFiles.forEach {
//use System.out directly instead of logger as we MUST output to screen due to user input requirement
System.out.println("#####${it.fileName}#####")
it.copyTo(System.out)
System.out.println("Is this output as expected? [y/N]")
validateUserResponse()
}
}
System.out.println("There is potential for data corruption.")
System.out.println("Are you sure that the migration scripts are acceptable? [y/N]")
validateUserResponse()
groupedBySourceJar.entries.forEach { (jar, migrationFiles) ->
val sourceJarFile = File(jar)
val migrationJarFile = File(sourceJarFile.parent, "migration-${sourceJarFile.name}")
JarOutputStream(FileOutputStream(migrationJarFile)).use { jos ->
migrationLogger.info("Creating migration cordapp at: ${migrationJarFile.absolutePath}")
migrationFiles.map { ZipEntry("migration/${it.fileName}") }.forEach {
jos.putNextEntry(it)
}
}
}
}
cmdLineOptions.createMigrationSqlForCordapp -> {
printWarning("The --$CREATE_MIGRATION_CORDAPP option has been deprecated and will be removed in a future version. Use the $CREATE_MIGRATION_CORDAPP command instead.")
createMigrationSqlForCordappsCli.cmdLineOptions.copyFrom(cmdLineOptions)
createMigrationSqlForCordappsCli.schemaClass = cmdLineOptions.createMigrationSqlForCordappPath!!
return createMigrationSqlForCordappsCli.runProgram()
}
else -> error("Please specify a correct command")
}
migrationLogger.info("No command specified.")
return ExitCodes.FAILURE
}
}
private fun validateUserResponse() {
val userInput = Scanner(System.`in`).nextLine().toLowerCase()
if (userInput != "y" && userInput != "yes") {
migrationLogger.warn("quitting due to user input")
System.exit(22)
}
data class Configuration(val dataSourceProperties: Properties, val database: DatabaseConfig, val jarDirs: List<String> = emptyList())
abstract class DbManagerConfiguration(private val cmdLineOptions: SharedDbManagerOptions) {
protected abstract val defaultConfigFileName: String
abstract val schemas: Set<MappedSchema>
abstract val classLoader: ClassLoader
abstract val parsedConfig: Config
val config by lazy { parsedConfig.parseAs(Configuration::class, UnknownConfigKeysPolicy.IGNORE::handle) }
val baseDirectory: Path by lazy {
val dir = cmdLineOptions.baseDirectory?.toAbsolutePath()?.normalize()
?: throw error("You must specify a base-directory")
if (!dir.exists()) throw error("Could not find base-directory: '${cmdLineOptions.baseDirectory}'.")
dir
}
private fun classLoaderFromJar(jarPath: Path): ClassLoader = URLClassLoader(listOf(jarPath.toUri().toURL()).toTypedArray())
private fun loadMappedSchema(schemaName: String, classLoader: ClassLoader) = classLoader.loadClass(schemaName).kotlin.objectInstance as MappedSchema
private fun getMigrationOutput(baseDirectory: Path): Writer {
return when (cmdLineOptions.dryRun) {
"" -> FileWriter(File(baseDirectory.toFile(), "migration${SimpleDateFormat("yyyyMMddHHmmss").format(Date())}.sql"))
CONSOLE -> PrintWriter(System.out)
else -> FileWriter(File(baseDirectory.toFile(), cmdLineOptions.dryRun))
}
}
private fun runWithDataSource(config: Configuration, baseDirectory: Path, classLoader: ClassLoader, withDatasource: (DataSource) -> Unit) {
val configFile by lazy { baseDirectory / (cmdLineOptions.configFile ?: (defaultConfigFileName)) }
fun runWithDataSource(withDatasource: (DataSource) -> Unit) {
val driversFolder = (baseDirectory / "drivers").let { if (it.exists()) listOf(it) else emptyList() }
val jarDirs = config.jarDirs.map { Paths.get(it) }
for (jarDir in jarDirs) {
@ -344,21 +132,36 @@ private class DbManagementTool : CordaCliWrapper("database-manager", "The Corda
}
}
private fun wrappedError(message: String, innerException: Exception) {
errorLogger.error(message, innerException)
throw WrappedConfigurationException(message, innerException)
}
private fun error(exception: Exception) {
errorLogger.error(exception.message, exception)
throw exception
}
private fun error(message: String) {
errorLogger.error(message)
throw ConfigurationException(message)
fun runMigrationCommand(schemas: Set<MappedSchema>, withMigration: (SchemaMigration, DataSource) -> Unit): Unit = this.runWithDataSource { dataSource ->
withMigration(SchemaMigration(schemas, dataSource, config.database, classLoader), dataSource)
}
}
class ConfigurationException(message: String) : Exception(message)
class WrappedConfigurationException(message: String, val innerException: Exception) : Exception(message)
class WrappedConfigurationException(message: String, val innerException: Exception) : Exception(message)
fun wrappedError(message: String, innerException: Exception) {
errorLogger.error(message, innerException)
throw WrappedConfigurationException(message, innerException)
}
fun error(exception: Exception) {
errorLogger.error(exception.message, exception)
throw exception
}
private fun error(message: String): Throwable {
errorLogger.error(message)
return ConfigurationException(message)
}
fun printInRed(message: String) {
println("${ShellConstants.RED}$message${ShellConstants.RESET}")
}
fun printWarning(message: String) {
Emoji.renderIfSupported {
printInRed("${Emoji.warningSign} ATTENTION: $message")
}
errorLogger.warn(message)
}

19
tools/dbmigration/src/main/kotlin/com/r3/corda/dbmigration/ReleaseLockCli.kt Normal file
View File

@ -0,0 +1,19 @@
package com.r3.corda.dbmigration
import net.corda.cliutils.CliWrapperBase
import net.corda.cliutils.ExitCodes
import net.corda.nodeapi.internal.persistence.SchemaMigration
import picocli.CommandLine
class ReleaseLockCli : CliWrapperBase(RELEASE_LOCK, "Releases whatever locks are on the database change log table, in case shutdown failed.") {
@CommandLine.Mixin
var cmdLineOptions = DbManagerOptions()
override fun runProgram(): Int {
val db = cmdLineOptions.toConfig()
db.runWithDataSource { it ->
SchemaMigration(emptySet(), it, db.config.database, Thread.currentThread().contextClassLoader).forceReleaseMigrationLock()
}
return ExitCodes.SUCCESS
}
}