[Python-modules-commits] [elasticsearch-curator] 02/06: Import elasticsearch-curator_3.4.0.orig.tar.gz
Apollon Oikonomopoulos
apoikos at moszumanska.debian.org
Wed Nov 4 12:39:52 UTC 2015
This is an automated email from the git hooks/post-receive script.
apoikos pushed a commit to tag debian/3.4.0-1
in repository elasticsearch-curator.
commit 76aeef48e56ce1899936c1d2a303ef058c7ba98e
Author: Apollon Oikonomopoulos <apoikos at debian.org>
Date: Wed Nov 4 14:28:44 2015 +0200
Import elasticsearch-curator_3.4.0.orig.tar.gz
---
CONTRIBUTORS | 2 +
Changelog.rst | 17 +
NOTICE | 96 ++++
curator/_version.py | 2 +-
curator/api/alias.py | 6 +-
curator/api/delete.py | 6 +
curator/api/filter.py | 14 +-
curator/api/utils.py | 48 +-
curator/cli/alias.py | 5 +-
curator/cli/allocation.py | 8 +-
curator/cli/cli.py | 10 +-
curator/cli/es_repo_mgr.py | 6 +-
curator/cli/index_selection.py | 39 +-
curator/cli/replicas.py | 5 +-
curator/cli/snapshot.py | 5 +-
curator/cli/snapshot_selection.py | 10 +-
curator/cli/utils.py | 22 +-
docs/Changelog.rst | 683 ++++++++++++++++++++++++++++-
docs/asciidoc/commands/allocation.asciidoc | 4 +-
docs/asciidoc/commands/optimize.asciidoc | 2 +-
docs/asciidoc/flags/index.asciidoc | 2 +
docs/asciidoc/flags/quiet.asciidoc | 32 ++
docs/asciidoc/index.asciidoc | 4 +-
docs/asciidoc/misc/installation.asciidoc | 20 +-
requirements.txt | 2 +-
setup.py | 2 +-
test/integration/__init__.py | 3 +
test/integration/test_api_utils.py | 24 +
test/integration/test_cli_commands.py | 18 +
test/unit/test_api_commands.py | 38 +-
test/unit/test_api_filter.py | 24 +-
test/unit/test_api_utils.py | 30 ++
test/unit/test_cli_utils.py | 2 +-
33 files changed, 1096 insertions(+), 95 deletions(-)
diff --git a/CONTRIBUTORS b/CONTRIBUTORS
index 360aefa..b00767b 100644
--- a/CONTRIBUTORS
+++ b/CONTRIBUTORS
@@ -62,3 +62,5 @@ Contributors:
* William Jimenez (wjimenez5271)
* Jeremy Canady (jrmycanady)
* Steven Ottenhoff (steffo)
+* Ole Rößner (Basster)
+* Jack (univerio)
diff --git a/Changelog.rst b/Changelog.rst
index 5f1e02c..da3420a 100644
--- a/Changelog.rst
+++ b/Changelog.rst
@@ -3,6 +3,23 @@
Changelog
=========
+3.4.0 (28 October 2015)
+-----------------------
+
+**General**
+
+ * API change in elasticsearch-py 1.7.0 prevented alias operations. Fixed in
+ #486 (HonzaKral)
+ * During index selection you can now select only closed indices with ``--closed-only``.
+ Does not impact ``--all-indices`` Reported in #476. Fixed in #487 (Basster)
+ * API Changes in Elasticsearch 2.0.0 required some refactoring. All tests pass
+ for ES versions 1.0.3 through 2.0.0-rc1. Fixed in #488 (untergeek)
+ * es_repo_mgr now has access to the same SSL options from #462. #489 (untergeek)
+ * Logging improvements requested in #475. (untergeek)
+ * Added ``--quiet`` flag. #494 (untergeek)
+ * Fixed ``index_closed`` to work with AWS Elasticsearch. #499 (univerio)
+ * Acceptable versions of Elasticsearch-py module are 1.8.0 up to 2.1.0 (untergeek)
+
3.3.0 (31 August 2015)
----------------------
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 0000000..712bbbd
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1,96 @@
+In accordance with section 4d of the Apache 2.0 license (http://www.apache.org/licenses/LICENSE-2.0),
+this NOTICE file is included.
+
+All users mentioned in the CONTRIBUTORS file at https://github.com/elastic/curator/blob/master/CONTRIBUTORS
+must be included in any derivative work.
+
+All conditions of section 4 of the Apache 2.0 license will be enforced:
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in
+ any medium, with or without modifications, and in Source or Object form, provided that You meet the
+ following conditions:
+
+a. You must give any other recipients of the Work or Derivative Works a copy of this License; and
+b. You must cause any modified files to carry prominent notices stating that You changed the files; and
+c. You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent,
+ trademark, and attribution notices from the Source form of the Work, excluding those notices that do
+ not pertain to any part of the Derivative Works; and
+d. If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that
+ You distribute must include a readable copy of the attribution notices contained within such NOTICE
+ file, excluding those notices that do not pertain to any part of the Derivative Works, in at least
+ one of the following places: within a NOTICE text file distributed as part of the Derivative Works;
+ within the Source form or documentation, if provided along with the Derivative Works; or, within a
+ display generated by the Derivative Works, if and wherever such third-party notices normally appear.
+ The contents of the NOTICE file are for informational purposes only and do not modify the License.
+ You may add Your own attribution notices within Derivative Works that You distribute, alongside or as
+ an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot
+ be construed as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and may provide additional or different
+ license terms and conditions for use, reproduction, or distribution of Your modifications, or for any
+ such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise
+ complies with the conditions stated in this License.
+
+ Contributors:
+ * Jordan Sissel (jordansissel) (For Logstash, first and foremost)
+ * Shay Banon (kimchy) (For Elasticsearch, of course!)
+ * Aaron Mildenstein (untergeek)
+ * Njal Karevoll
+ * François Deppierraz
+ * Honza Kral (HonzaKral)
+ * Benjamin Smith (benjaminws)
+ * Colin Moller (LeftyBC)
+ * Elliot (edgeofnite)
+ * Ram Viswanadha (ramv)
+ * Chris Meisinger (cmeisinger)
+ * Stuart Warren (stuart-warren)
+ * (gitshaw)
+ * (sfritz)
+ * (sjoelsam)
+ * Jose Diaz-Gonzalez (josegonzalez)
+ * Arie Bro (arieb)
+ * David Harrigan (dharrigan)
+ * Mathieu Geli (gelim)
+ * Nick Ethier (nickethier)
+ * Mohab Usama (mohabusama)
+ * (gitshaw)
+ * Stuart Warren (stuart-warren)
+ * Xavier Calland (xavier-calland)
+ * Chad Schellenger (cschellenger)
+ * Kamil Essekkat (ekamil)
+ * (gbutt)
+ * Ben Buchacher (bbuchacher)
+ * Ehtesh Choudhury (shurane)
+ * Markus Fischer (mfn)
+ * Fabien Wernli (faxm0dem)
+ * Michael Weiser (michaelweiser)
+ * (digital-wonderland)
+ * cassiano (cassianoleal)
+ * Matt Dainty (bodgit)
+ * Alex Philipp (alex-sf)
+ * (krzaczek)
+ * Justin Lintz (jlintz)
+ * Jeremy Falling (jjfalling)
+ * Ian Babrou (bobrik)
+ * Ferenc Erki (ferki)
+ * George Heppner (gheppner)
+ * Matt Hughes (matthughes)
+ * Brian Lalor (blalor)
+ * Paweł Krzaczkowski (krzaczek)
+ * Ben Tse (bt5e)
+ * Tom Hendrikx (whyscream)
+ * Christian Vozar (christianvozar)
+ * Magnus Baeck (magnusbaeck)
+ * Robin Kearney (rk295)
+ * (cfeio)
+ * (malagoli)
+ * Dan Sheridan (djs52)
+ * Michael-Keith Bernard (SegFaultAX)
+ * Simon Lundström (simmel)
+ * (pkr1234)
+ * Mark Feltner (feltnerm)
+ * William Jimenez (wjimenez5271)
+ * Jeremy Canady (jrmycanady)
+ * Steven Ottenhoff (steffo)
+ * Ole Rößner (Basster)
+ * Jack (univerio)
diff --git a/curator/_version.py b/curator/_version.py
index 6a157dc..f631007 100644
--- a/curator/_version.py
+++ b/curator/_version.py
@@ -1 +1 @@
-__version__ = '3.3.0'
+__version__ = '3.4.0'
diff --git a/curator/api/alias.py b/curator/api/alias.py
index 233e707..1f6a0a4 100644
--- a/curator/api/alias.py
+++ b/curator/api/alias.py
@@ -21,7 +21,7 @@ def add_to_alias(client, index_name, alias=None):
if not alias: # This prevents _all from being aliased by accident...
logger.error('No alias provided.')
return False
- if not client.indices.exists_alias(alias):
+ if not client.indices.exists_alias(name=alias):
indices_in_alias = []
logger.info('Alias {0} not found. Creating...'.format(alias))
else:
@@ -33,12 +33,13 @@ def add_to_alias(client, index_name, alias=None):
else:
try:
client.indices.update_aliases(body={'actions': [{ 'add': { 'index': index_name, 'alias': alias}}]})
+ logger.info('Adding index {0} to alias {1}...'.format(index_name, alias))
return True
except Exception as e:
logger.error("Error adding index {0} to alias {1}. Exception: {2} Run with --debug flag and/or check Elasticsearch logs for more information.".format(index_name, alias, e))
return False
else:
- logger.info('Skipping index {0}: Index already exists in alias {1}...'.format(index_name, alias))
+ logger.debug('Skipping index {0}: Index already exists in alias {1}...'.format(index_name, alias))
return True
def remove_from_alias(client, index_name, alias=None):
@@ -63,6 +64,7 @@ def remove_from_alias(client, index_name, alias=None):
if index_name in indices_in_alias:
try:
client.indices.update_aliases(body={'actions': [{ 'remove': { 'index': index_name, 'alias': alias}}]})
+ logger.info("Removing index {0} from alias {1}.".format(index_name, alias))
return True
except Exception as e:
logger.error("Error removing index {0} from alias {1}. Exception: {2} Run with --debug flag and/or check Elasticsearch logs for more information.".format(index_name, alias, e))
diff --git a/curator/api/delete.py b/curator/api/delete.py
index be07586..e4985e0 100644
--- a/curator/api/delete.py
+++ b/curator/api/delete.py
@@ -14,6 +14,9 @@ def delete_indices(client, indices, master_timeout=30000):
:rtype bool:
"""
indices = ensure_list(indices)
+ if get_version(client) >= (2,0,0):
+ if type(master_timeout) == type(int()):
+ master_timeout = str(master_timeout/1000) + 's'
try:
logger.info("Deleting indices as a batch operation:")
for i in indices:
@@ -37,6 +40,9 @@ def delete(client, indices, master_timeout=30000):
:arg master_timeout: Number of milliseconds to wait for master node response
:rtype bool:
"""
+ if get_version(client) >= (2,0,0):
+ if type(master_timeout) == type(int()):
+ master_timeout = str(master_timeout/1000) + 's'
for count in range(1, 4): # Try 3 times
logger.debug("master_timeout value: {0}".format(master_timeout))
delete_indices(client, indices, master_timeout)
diff --git a/curator/api/filter.py b/curator/api/filter.py
index 19bf130..ab2dc0d 100644
--- a/curator/api/filter.py
+++ b/curator/api/filter.py
@@ -324,11 +324,10 @@ def filter_by_space(client, indices, disk_space=None, reverse=True):
def get_stat_list(stats):
retval = []
- for (index_name, index_stats) in stats['indices'].items():
- replica_count = client.indices.get_settings(index_name)[index_name]['settings']['index']['number_of_replicas']
- size = "size_in_bytes" if int(replica_count) > 0 else "primary_size_in_bytes"
- logger.debug('Index: {0} Size: {1}'.format(index_name, index_stats['index'][size]))
- retval.append((index_name, index_stats['index'][size]))
+ for index_name in stats['indices']:
+ size = stats['indices'][index_name]['total']['store']['size_in_bytes']
+ logger.debug('Index: {0} Size: {1}'.format(index_name, size))
+ retval.append((index_name, size))
return retval
# Ensure that disk_space is a float
@@ -355,9 +354,9 @@ def filter_by_space(client, indices, disk_space=None, reverse=True):
index_lists = chunk_index_list(not_closed)
statlist = []
for l in index_lists:
- statlist.extend(get_stat_list(client.indices.status(index=to_csv(l))))
+ statlist.extend(get_stat_list(client.indices.stats(index=to_csv(l))))
else:
- statlist = get_stat_list(client.indices.status(index=to_csv(not_closed)))
+ statlist = get_stat_list(client.indices.stats(index=to_csv(not_closed)))
sorted_indices = sorted(statlist, reverse=reverse)
@@ -366,6 +365,7 @@ def filter_by_space(client, indices, disk_space=None, reverse=True):
if disk_usage > disk_limit:
delete_list.append(index_name)
+ logger.info('Deleting {0}, summed disk usage is {1:.3f} GB and disk limit is {2:.3f} GB.'.format(index_name, disk_usage/2**30, disk_limit/2**30))
else:
logger.info('skipping {0}, summed disk usage is {1:.3f} GB and disk limit is {2:.3f} GB.'.format(index_name, disk_usage/2**30, disk_limit/2**30))
return delete_list
diff --git a/curator/api/utils.py b/curator/api/utils.py
index 9fafece..7ccd277 100644
--- a/curator/api/utils.py
+++ b/curator/api/utils.py
@@ -4,6 +4,7 @@ import time
import re
import sys
import logging
+import json
logger = logging.getLogger(__name__)
def get_alias(client, alias):
@@ -14,7 +15,7 @@ def get_alias(client, alias):
:arg alias: Alias name to operate on.
:rtype: list of strings
"""
- if client.indices.exists_alias(alias):
+ if client.indices.exists_alias(name=alias):
return client.indices.get_alias(name=alias).keys()
else:
logger.error('Unable to find alias {0}.'.format(alias))
@@ -88,11 +89,23 @@ def index_closed(client, index_name):
:arg index_name: The index name
:rtype: bool
"""
- index_metadata = client.cluster.state(
- index=index_name,
- metric='metadata',
- )
- return index_metadata['metadata']['indices'][index_name]['state'] == 'close'
+ # This workaround using the _cat API is for AWS Elasticsearch, since it does
+ # not allow users to poll the cluster state. It may also be faster than
+ # using the cluster state.
+ if get_version(client) >= (1, 5, 0):
+ indices = client.cat.indices(index=index_name, format='json', h='status')
+ # This workaround is also for AWS, as the _cat API still returns text/plain
+ # even with ``format='json'``.
+ if not isinstance(indices, list): # content-type: text/plain
+ indices = json.loads(indices)
+ (index_info,) = indices
+ return index_info['status'] == 'close'
+ else:
+ index_metadata = client.cluster.state(
+ index=index_name,
+ metric='metadata',
+ )
+ return index_metadata['metadata']['indices'][index_name]['state'] == 'close'
def get_segmentcount(client, index_name):
"""
@@ -155,9 +168,10 @@ def optimized(client, index_name, max_num_segments=None):
shards, segmentcount = get_segmentcount(client, index_name)
logger.debug('Index {0} has {1} shards and {2} segments total.'.format(index_name, shards, segmentcount))
if segmentcount > (shards * max_num_segments):
+ logger.debug('Flagging index {0} for optimization.'.format(index_name))
return False
else:
- logger.info('Skipping index {0}: Already optimized.'.format(index_name))
+ logger.debug('Skipping index {0}: Already optimized.'.format(index_name))
return True
def get_version(client):
@@ -292,13 +306,28 @@ def prune_closed(client, indices):
:arg indices: A list of indices to act on
:rtype: list
"""
+ return prune_open_or_closed(client, indices, append_closed=False)
+
+def prune_opened(client, indices):
+ """
+ Return list of indices that are not open.
+
+ :arg client: The Elasticsearch client connection
+ :arg indices: A list of indices to act on
+ :rtype: list
+ """
+ return prune_open_or_closed(client, indices, append_closed=True)
+
+def prune_open_or_closed(client, indices, append_closed):
indices = ensure_list(indices)
retval = []
+ status = 'Closed' if append_closed else 'Opened'
for idx in list(indices):
- if not index_closed(client, idx):
+ if index_closed(client, idx) is append_closed:
retval.append(idx)
+ logger.debug('Including index {0}: {1}.'.format(idx, status))
else:
- logger.info('Skipping index {0}: Closed.'.format(idx))
+ logger.debug('Skipping index {0}: {1}.'.format(idx, status))
return sorted(retval)
def prune_allocated(client, indices, key, value, allocation_type):
@@ -326,5 +355,6 @@ def prune_allocated(client, indices, key, value, allocation_type):
if has_routing:
logger.debug('Skipping index {0}: Allocation rule {1} is already applied for type {2}.'.format(idx, key + "=" + value, allocation_type))
else:
+ logger.info('Flagging index {0} to have allocation rule {1} applied for type {2}.'.format(idx, key + "=" + value, allocation_type))
retval.append(idx)
return sorted(retval)
diff --git a/curator/cli/alias.py b/curator/cli/alias.py
index 0adbb27..4756be4 100644
--- a/curator/cli/alias.py
+++ b/curator/cli/alias.py
@@ -12,7 +12,8 @@ logger = logging.getLogger(__name__)
def alias(ctx, name, remove):
"""Index Aliasing"""
if not name:
- click.echo('{0}'.format(ctx.get_help()))
- click.echo(click.style('Missing required parameter --name', fg='red', bold=True))
+ msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet'])
+ logger.error('Missing required parameter --name')
+ msgout('Missing required parameter --name', error=True, quiet=ctx.parent.params['quiet'])
sys.exit(1)
alias.add_command(indices)
diff --git a/curator/cli/allocation.py b/curator/cli/allocation.py
index c9b9c9f..9e7e91b 100644
--- a/curator/cli/allocation.py
+++ b/curator/cli/allocation.py
@@ -14,12 +14,14 @@ logger = logging.getLogger(__name__)
def allocation(ctx, rule, type):
"""Index Allocation"""
if not rule:
- click.echo('{0}'.format(ctx.get_help()))
- click.echo(click.style('Missing required parameter --rule', fg='red', bold=True))
+ msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet'])
+ logger.error('Missing required parameter --rule')
+ msgout('Missing required parameter --rule', error=True, quiet=ctx.parent.params['quiet'])
sys.exit(1)
if type not in ['require', 'include', 'exclude']:
- click.echo(click.style('--type can only be one of: require, include exclude', fg='red', bold=True))
+ logger.error('--type can only be one of: require, include exclude')
+ msgout('--type can only be one of: require, include exclude', error=True, quiet=ctx.parent.params['quiet'])
sys.exit(1)
allocation.add_command(indices)
diff --git a/curator/cli/cli.py b/curator/cli/cli.py
index 65ae2ff..43563b4 100644
--- a/curator/cli/cli.py
+++ b/curator/cli/cli.py
@@ -44,9 +44,10 @@ DEFAULT_ARGS = {
@click.option('--loglevel', help='Log level', default=DEFAULT_ARGS['log_level'])
@click.option('--logfile', help='log file')
@click.option('--logformat', help='Log output format [default|logstash].', default=DEFAULT_ARGS['logformat'])
+ at click.option('--quiet', help='Suppress command-line output.', is_flag=True)
@click.version_option(version=__version__)
@click.pass_context
-def cli(ctx, host, url_prefix, port, use_ssl, certificate, ssl_no_validate, http_auth, timeout, master_only, dry_run, debug, loglevel, logfile, logformat):
+def cli(ctx, host, url_prefix, port, use_ssl, certificate, ssl_no_validate, http_auth, timeout, master_only, dry_run, debug, loglevel, logfile, logformat, quiet):
"""
Curator for Elasticsearch indices.
@@ -55,7 +56,7 @@ def cli(ctx, host, url_prefix, port, use_ssl, certificate, ssl_no_validate, http
# Setup logging
if debug:
- numeric_log_level = logging.DEBUG
+ numeric_log_level = logging.DEBUG or loglevel.upper() == 'DEBUG'
format_string = '%(asctime)s %(levelname)-9s %(name)22s %(funcName)22s:%(lineno)-4d %(message)s'
else:
numeric_log_level = getattr(logging, loglevel.upper(), None)
@@ -66,18 +67,21 @@ def cli(ctx, host, url_prefix, port, use_ssl, certificate, ssl_no_validate, http
handler = logging.StreamHandler(
open(logfile, 'a') if logfile else sys.stdout)
if logformat == 'logstash':
+ ctx.params['quiet'] = True
handler.setFormatter(LogstashFormatter())
else:
handler.setFormatter(logging.Formatter(format_string))
logging.root.addHandler(handler)
logging.root.setLevel(numeric_log_level)
+ logger = logging.getLogger('curator.cli')
# Test whether certificate is a valid file path
if use_ssl is True and certificate is not None:
try:
open(certificate, 'r')
except IOError:
- click.echo(click.style('Error: Could not open certificate at {0}'.format(certificate), fg='red', bold=True))
+ logger.error('Could not open certificate at {0}'.format(certificate))
+ msgout('Error: Could not open certificate at {0}'.format(certificate), error=True, quiet=quiet)
sys.exit(1)
# Filter out logging from Elasticsearch and associated modules by default
diff --git a/curator/cli/es_repo_mgr.py b/curator/cli/es_repo_mgr.py
index 71e47b0..97f9a1d 100644
--- a/curator/cli/es_repo_mgr.py
+++ b/curator/cli/es_repo_mgr.py
@@ -188,6 +188,8 @@ def s3(
@click.option('--url_prefix', help='Elasticsearch http url prefix.', default=DEFAULT_ARGS['url_prefix'])
@click.option('--port', help='Elasticsearch port.', default=DEFAULT_ARGS['port'], type=int)
@click.option('--use_ssl', help='Connect to Elasticsearch through SSL.', is_flag=True, default=DEFAULT_ARGS['use_ssl'])
+ at click.option('--certificate', help='Path to certificate to use for SSL validation. (OPTIONAL)', type=str, default=None)
+ at click.option('--ssl-no-validate', help='Do not validate SSL certificate', is_flag=True)
@click.option('--http_auth', help='Use Basic Authentication ex: user:pass', default=DEFAULT_ARGS['http_auth'])
@click.option('--timeout', help='Connection timeout in seconds.', default=DEFAULT_ARGS['timeout'], type=int)
@click.option('--master-only', is_flag=True, help='Only operate on elected master node.')
@@ -197,13 +199,13 @@ def s3(
@click.option('--logformat', help='Log output format [default|logstash].', default=DEFAULT_ARGS['logformat'])
@click.version_option(version=__version__)
@click.pass_context
-def repomgrcli(ctx, host, url_prefix, port, use_ssl, http_auth, timeout, master_only, debug, loglevel, logfile, logformat):
+def repomgrcli(ctx, host, url_prefix, port, use_ssl, certificate, ssl_no_validate, http_auth, timeout, master_only, debug, loglevel, logfile, logformat):
"""Repository manager for Elasticsearch Curator.
"""
# Setup logging
if debug:
- numeric_log_level = logging.DEBUG
+ numeric_log_level = logging.DEBUG or loglevel.upper() == 'DEBUG'
format_string = '%(asctime)s %(levelname)-9s %(name)22s %(funcName)22s:%(lineno)-4d %(message)s'
else:
numeric_log_level = getattr(logging, loglevel.upper(), None)
diff --git a/curator/cli/index_selection.py b/curator/cli/index_selection.py
index 6b7d630..7b649ce 100644
--- a/curator/cli/index_selection.py
+++ b/curator/cli/index_selection.py
@@ -30,9 +30,11 @@ logger = logging.getLogger(__name__)
help='Include the provided index in the list. Can be invoked multiple times.')
@click.option('--all-indices', is_flag=True,
help='Do not filter indices. Act on all indices.')
+ at click.option('--closed-only', is_flag=True,
+ help='Include only indices that are closed.')
@click.pass_context
def indices(ctx, newer_than, older_than, prefix, suffix, time_unit,
- timestring, regex, exclude, index, all_indices):
+ timestring, regex, exclude, index, all_indices, closed_only):
"""
Get a list of indices to act on from the provided arguments, then perform
the command [alias, allocation, bloom, close, delete, etc.] on the resulting
@@ -46,8 +48,10 @@ def indices(ctx, newer_than, older_than, prefix, suffix, time_unit,
regex = r'^.*{0}.*$'.format(get_date_regex(timestring))
ctx.obj['filters'].append({ 'pattern': regex })
if not all_indices and not ctx.obj['filters'] and not index:
- click.echo('{0}'.format(ctx.get_help()))
- click.echo(click.style('ERROR. At least one filter must be supplied.', fg='red', bold=True))
+
+ logger.error('At least one filter must be supplied.')
+ msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.parent.params['quiet'])
+ msgout('ERROR. At least one filter must be supplied.', error=True, quiet=ctx.parent.parent.params['quiet'])
sys.exit(1)
logger.info("Job starting: {0} indices".format(ctx.parent.info_name))
@@ -78,9 +82,14 @@ def indices(ctx, newer_than, older_than, prefix, suffix, time_unit,
if indices:
working_list = indices
else:
- click.echo(click.style('ERROR. No indices found in Elasticsearch.', fg='red', bold=True))
+ logger.error('No indices found in Elasticsearch')
+ msgout('ERROR. No indices found in Elasticsearch.', error=True, quiet=ctx.parent.parent.params['quiet'])
sys.exit(1)
+ if closed_only and not all_indices:
+ logger.info("Pruning open indices, leaving only closed indices.")
+ working_list = prune_opened(client, working_list)
+
# Override any other flags if --all_indices specified
if all_indices:
working_list = indices
@@ -104,6 +113,7 @@ def indices(ctx, newer_than, older_than, prefix, suffix, time_unit,
if working_list and ctx.parent.info_name == 'delete':
# If filter by disk space, filter the working_list by space:
if ctx.parent.params['disk_space']:
+ logger.info("Filtering to keep disk usage below {0} gigabytes".format(ctx.parent.params['disk_space']))
working_list = filter_by_space(
client, working_list,
disk_space=ctx.parent.params['disk_space'],
@@ -116,13 +126,22 @@ def indices(ctx, newer_than, older_than, prefix, suffix, time_unit,
if timestring and not newer_than and not older_than \
and not (ctx.parent.info_name == 'show') \
and not ctx.parent.parent.params['dry_run']:
- click.echo(click.style('You are using --timestring without --older-than or --newer-than.', fg='yellow', bold=True))
- click.echo('This could result in actions being performed on all indices matching {0}'.format(timestring))
- click.echo(click.style('Press CTRL-C to exit Curator before the timer expires:', fg='red', bold=True))
- countdown(10)
+ if ctx.parent.parent.params['quiet']:
+ # Don't output to stdout if 'quiet' (or logformat == logstash)
+ logger.warn('You are using --timestring without --older-than or --newer-than.')
+ logger.warn('This could result in actions being performed on all indices matching {0}'.format(timestring))
+ else:
+ # Do this if not quiet mode.
+ logger.warn('You are using --timestring without --older-than or --newer-than.')
+ logger.warn('This could result in actions being performed on all indices matching {0}'.format(timestring))
+ msgout('You are using --timestring without --older-than or --newer-than.', warning=True, quiet=ctx.parent.parent.params['quiet'])
+ msgout('This could result in actions being performed on all indices matching {0}'.format(timestring), warning=True, quiet=ctx.parent.parent.params['quiet'])
+ msgout('Press CTRL-C to exit Curator before the timer expires:', error=True, quiet=ctx.parent.parent.params['quiet'])
+ countdown(10)
+
# Make a sorted, unique list of indices
working_list = sorted(list(set(working_list)))
- logger.debug('ACTION: {0}. INDICES: {1}'.format(ctx.parent.info_name, working_list))
+ logger.info('Action {0} will be performed on the following indices: {1}'.format(ctx.parent.info_name, working_list))
# Do action here!!! Don't forget to account for DRY_RUN!!!
if ctx.parent.info_name == 'show':
@@ -150,5 +169,5 @@ def indices(ctx, newer_than, older_than, prefix, suffix, time_unit,
else:
logger.warn('No indices matched provided args: {0}'.format(ctx.params))
- click.echo(click.style('No indices matched provided args.', fg='red', bold=True))
+ msgout('No indices matched provided args: {0}'.format(ctx.params), quiet=ctx.parent.parent.params['quiet'])
sys.exit(0)
diff --git a/curator/cli/replicas.py b/curator/cli/replicas.py
index 1be38d3..0c21c63 100644
--- a/curator/cli/replicas.py
+++ b/curator/cli/replicas.py
@@ -11,7 +11,8 @@ logger = logging.getLogger(__name__)
def replicas(ctx, count):
"""Replica Count Per-shard"""
if count == None: # Have to do this since 0 is valid
- click.echo('{0}'.format(ctx.get_help()))
- click.echo(click.style('Missing required parameter --count', fg='red', bold=True))
+ msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet'])
+ logger.error('Missing required parameter --count')
+ msgout('Missing required parameter --count', error=True, quiet=ctx.parent.params['quiet'])
sys.exit(1)
replicas.add_command(indices)
diff --git a/curator/cli/snapshot.py b/curator/cli/snapshot.py
index 80afd86..6ae7ed2 100644
--- a/curator/cli/snapshot.py
+++ b/curator/cli/snapshot.py
@@ -38,7 +38,8 @@ def snapshot(
):
"""Take snapshots of indices (Backup)"""
if not repository:
- click.echo('{0}'.format(ctx.get_help()))
- click.echo(click.style('Missing required parameter --repository', fg='red', bold=True))
+ msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.params['quiet'])
+ logger.error('Missing required parameter --repository')
+ msgout('Missing required parameter --repository', error=True, quiet=ctx.parent.params['quiet'])
sys.exit(1)
snapshot.add_command(indices)
diff --git a/curator/cli/snapshot_selection.py b/curator/cli/snapshot_selection.py
index 4522b07..f91565a 100644
--- a/curator/cli/snapshot_selection.py
+++ b/curator/cli/snapshot_selection.py
@@ -43,8 +43,9 @@ def snapshots(ctx, newer_than, older_than, prefix, suffix, time_unit,
"""
if not repository:
- click.echo('{0}'.format(ctx.get_help()))
- click.echo(click.style('Missing required --repository parameter.', fg='red', bold=True))
+ msgout('{0}'.format(ctx.get_help()), quiet=ctx.parent.parent.params['quiet'])
+ logger.error('Missing required parameter --repository')
+ msgout('Missing required parameter --repository', error=True, quiet=ctx.parent.parent.params['quiet'])
sys.exit(1)
logger.info("Job starting: {0} snapshots".format(ctx.parent.info_name))
@@ -59,7 +60,8 @@ def snapshots(ctx, newer_than, older_than, prefix, suffix, time_unit,
if snapshots:
working_list = snapshots
else:
- click.echo(click.style('No snapshots found in Elasticsearch.', fg='red', bold=True))
+ logger.error('No snapshots found in Elasticsearch.')
+ msgout('No snapshots found in Elasticsearch.', error=True, quiet=ctx.parent.parent.params['quiet'])
sys.exit(0)
if all_snapshots:
@@ -96,5 +98,5 @@ def snapshots(ctx, newer_than, older_than, prefix, suffix, time_unit,
else:
logger.warn('No snapshots matched provided args.')
- click.echo(click.style('No snapshots matched provided args.', fg='red', bold=True))
+ msgout('No snapshots matched provided args.', quiet=ctx.parent.parent.params['quiet'])
sys.exit(0)
diff --git a/curator/cli/utils.py b/curator/cli/utils.py
index 5e7b785..1c0bf03 100644
--- a/curator/cli/utils.py
+++ b/curator/cli/utils.py
@@ -12,7 +12,7 @@ from ..api import *
logger = logging.getLogger(__name__)
# Elasticsearch versions supported
-version_max = (2, 0, 0)
+version_max = (3, 0, 0)
version_min = (1, 0, 0)
REGEX_MAP = {
@@ -89,8 +89,8 @@ def check_version(client):
version_number = get_version(client)
logger.debug('Detected Elasticsearch version {0}'.format(".".join(map(str,version_number))))
if version_number >= version_max or version_number < version_min:
- click.echo(click.style('Expected Elasticsearch version range > {0} < {1}'.format(".".join(map(str,version_min)),".".join(map(str,version_max))), fg='red'))
- click.echo(click.style('ERROR: Incompatible with version {0} of Elasticsearch. Exiting.'.format(".".join(map(str,version_number))), fg='red', bold=True))
+ logger.error('Expected Elasticsearch version range > {0} < {1}'.format(".".join(map(str,version_min)),".".join(map(str,version_max))))
+ logger.error('Incompatible with version {0} of Elasticsearch.'.format(".".join(map(str,version_number))))
sys.exit(1)
def check_master(client, master_only=False):
@@ -135,7 +135,7 @@ def get_client(**kwargs):
check_master(client, master_only=master_only)
return client
except Exception:
- click.echo(click.style('ERROR: Connection failure.', fg='red', bold=True))
+ logger.error('Connection failure.')
sys.exit(1)
def override_timeout(ctx):
@@ -163,10 +163,10 @@ def filter_callback(ctx, param, value):
if param.name in ['older_than', 'newer_than']:
if not ctx.params['time_unit'] :
- click.echo(click.style("Parameters --older-than and --newer-than require the --time-unit parameter", fg='red', bold=True))
+ logger.error("Parameters --older-than and --newer-than require the --time-unit parameter")
sys.exit(1)
if not ctx.params['timestring']:
- click.echo(click.style("Parameters --older-than and --newer-than require the --timestring parameter", fg='red', bold=True))
+ logger.error("Parameters --older-than and --newer-than require the --timestring parameter")
sys.exit(1)
argdict = { "groupname":'date', "time_unit":ctx.params["time_unit"],
"timestring": ctx.params['timestring'], "value": value,
@@ -249,3 +249,13 @@ def do_command(client, command, indices, params=None, master_timeout=30000):
request_timeout=params['request_timeout'],
skip_repo_validation=params['skip_repo_validation'],
)
+
+def msgout(msg, error=False, warning=False, quiet=False):
+ """Output messages to stdout via click.echo if quiet=False"""
+ if not quiet:
+ if error:
+ click.echo(click.style(click.style(msg, fg='red', bold=True)))
+ elif warning:
+ click.echo(click.style(click.style(msg, fg='yellow', bold=True)))
+ else:
+ click.echo(msg)
diff --git a/docs/Changelog.rst b/docs/Changelog.rst
deleted file mode 120000
index bd5ae16..0000000
--- a/docs/Changelog.rst
+++ /dev/null
@@ -1 +0,0 @@
-../Changelog.rst
\ No newline at end of file
diff --git a/docs/Changelog.rst b/docs/Changelog.rst
new file mode 100644
index 0000000..09683c6
--- /dev/null
+++ b/docs/Changelog.rst
@@ -0,0 +1,682 @@
+.. _changelog:
+
+Changelog
+=========
+
+3.4.0 (28 October 2015)
+-----------------------
+
+**General**
+
+ * API change in elasticsearch-py 1.7.0 prevented alias operations. Fixed in
+ #486 (HonzaKral)
+ * During index selection you can now select only closed indices with ``--closed-only``.
+ Does not impact ``--all-indices`` Reported in #476. Fixed in #487 (Basster)
+ * API Changes in Elasticsearch 2.0.0 required some refactoring. All tests pass
+ for ES versions 1.0.3 through 2.0.0-rc1. Fixed in #488 (untergeek)
+ * es_repo_mgr now has access to the same SSL options from #462. #489 (untergeek)
+ * Logging improvements requested in #475. (untergeek)
+ * Added ``--quiet`` flag. #494 (untergeek)
+ * Fixed ``index_closed`` to work with AWS Elasticsearch. #499 (univerio)
+
+3.3.0 (31 August 2015)
+----------------------
+
+**Announcement**
+
+ * Curator is tested in Jenkins. Each commit to the master branch is tested
+ with both Python versions 2.7.6 and 3.4.0 against each of the following
+ Elasticsearch versions:
+ * 1.7_nightly
+ * 1.6_nightly
+ * 1.7.0
+ * 1.6.1
+ * 1.5.1
+ * 1.4.4
+ * 1.3.9
+ * 1.2.4
+ * 1.1.2
+ * 1.0.3
+ * If you are using a version different from this, your results may vary.
+
+**General**
+
+ * Allocation type can now also be ``include`` or ``exclude``, in addition to the
+ the existing default ``require`` type. Add ``--type`` to the allocation command
+ to specify the type. #443 (steffo)
+
+ * Bump elasticsearch python module dependency to 1.6.0+ to enable synced_flush
+ API call. Reported in #447 (untergeek)
+
+ * Add SSL features, ``--ssl-no-validate`` and ``certificate`` to provide other
+ ways to validate SSL connections to Elasticsearch. #436 (untergeek)
+
+**Bug fixes**
+
+ * Delete by space was only reporting space used by primary shards. Fixed to
+ show all space consumed. Reported in #455 (untergeek)
+
+ * Update exit codes and messages for snapshot selection. Reported in #452 (untergeek)
+
+ * Fix potential int/float casting issues. Reported in #465 (untergeek)
+
+3.2.3 (16 July 2015)
+--------------------
+
+**Bug fix**
+
+ * In order to address customer and community issues with bulk deletes, the
+ ``master_timeout`` is now invoked for delete operations. This should address
+ 503s with 30s timeouts in the debug log, even when ``--timeout`` is set to
+ a much higher value. The ``master_timeout`` is tied to the ``--timeout``
+ flag value, but will not exceed 300 seconds. #420 (untergeek)
+
+**General**
+
+ * Mixing it up a bit here by putting `General` second! The only other changes
+ are that logging has been improved for deletes so you won't need to have the
+ ``--debug`` flag to see if you have error codes >= 400, and some code
+ documentation improvements.
+
+3.2.2 (13 July 2015)
+--------------------
+
+**General**
+
+ * This is a very minor change. The ``mock`` library recently removed support
+ for Python 2.6. As many Curator users are using RHEL/CentOS 6, which is
+ pinned to Python 2.6, this requires the mock version referenced by Curator
+ to also be pinned to a supported version (``mock==1.0.1``).
+
+3.2.1 (10 July 2015)
+--------------------
+
+**General**
+
+ * Added delete verification & retry (fixed at 3x) to potentially cover an edge
+ case in #420 (untergeek)
+ * Since GitHub allows rST (reStructuredText) README documents, and that's what
+ PyPI wants also, the README has been rebuilt in rST. (untergeek)
+
+**Bug fixes**
+
+ * If closing indices with ES 1.6+, and all indices are closed, ensure that the
+ seal command does not try to seal all indices. Reported in #426 (untergeek)
+ * Capture AttributeError when sealing indices if a non-TransportError occurs.
+ Reported in #429 (untergeek)
+
+3.2.0 (25 June 2015)
+--------------------
+
+**New!**
+
+ * Added support to manually seal, or perform a [synced flush](http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-synced-flush.html)
+ on indices with the ``seal`` command. #394 (untergeek)
+ * Added *experimental* support for SSL certificate validation. In order for
+ this to work, you must install the ``certifi`` python module:
+ ``pip install certifi``
+ This feature *should* automatically work if the ``certifi`` module is
+ installed. Please report any issues.
+
+**General**
+
+ * Changed logging to go to stdout rather than stderr. Reopened #121 and
+ figured they were right. This is better. (untergeek)
+ * Exit code 99 was unpopular. It has been removed. Reported in #371 and #391
+ (untergeek)
+ * Add ``--skip-repo-validation`` flag for snapshots. Do not validate write
+ access to repository on all cluster nodes before proceeding. Useful for
+ shared filesystems where intermittent timeouts can affect validation, but
+ won't likely affect snapshot success. Requested in #396 (untergeek)
+ * An alias no longer needs to be pre-existent in order to use the alias
+ command. #317 (untergeek)
+ * es_repo_mgr now passes through upstream errors in the event a repository
+ fails to be created. Requested in #405 (untergeek)
+
+**Bug fixes**
+
+ * In rare cases, ``*`` wildcard would not expand. Replaced with _all.
+ Reported in #399 (untergeek)
+ * Beginning with Elasticsearch 1.6, closed indices cannot have their replica
+ count altered. Attempting to do so results in this error:
+ ``org.elasticsearch.ElasticsearchIllegalArgumentException: Can't update [index.number_of_replicas] on closed indices [[test_index]] - can leave index in an unopenable state``
+ As a result, the ``change_replicas`` method has been updated to prune closed
+ indices. This change will apply to all versions of Elasticsearch.
+ Reported in #400 (untergeek)
+ * Fixed es_repo_mgr repository creation verification error. Reported in #389
+ (untergeek)
+
+
+
+3.1.0 (21 May 2015)
+-------------------
+
+**General**
+
+ * If ``wait_for_completion`` is true, snapshot success is now tested and logged.
+ Reported in #253 (untergeek)
+ * Log & return false if a snapshot is already in progress (untergeek)
+ * Logs individual deletes per index, even though they happen in batch mode.
+ Also log individual snapshot deletions. Reported in #372 (untergeek)
+ * Moved ``chunk_index_list`` from cli to api utils as it's now also used by ``filter.py``
+ * Added a warning and 10 second timer countdown if you use ``--timestring`` to filter
+ indices, but do not use ``--older-than`` or ``--newer-than`` in conjunction with it.
+ This is to address #348, which behavior isn't a bug, but prevents accidental
+ action against all of your time-series indices. The warning and timer are
+ not displayed for ``show`` and ``--dry-run`` operations.
+ * Added tests for ``es_repo_mgr`` in #350
+ * Doc fixes
+
+**Bug fixes**
+
+ * delete-by-space needed the same fix used for #245. Fixed in #353 (untergeek)
+ * Increase default client timeout for ``es_repo_mgr`` as node discovery and
+ availability checks for S3 repositories can take a bit. Fixed in #352 (untergeek)
+ * If an index is closed, indicate in ``show`` and ``--dry-run`` output.
+ Reported in #327. (untergeek)
+ * Fix issue where CLI parameters were not being passed to the ``es_repo_mgr``
+ create sub-command.
+ Reported in #337. (feltnerm)
+
+3.0.3 (27 Mar 2015)
+-------------------
+
+**Announcement**
+
+This is a bug fix release. #319 and #320 are affecting a few users, so this
+release is being expedited.
+
+Test count: 228
+Code coverage: 99%
+
+**General**
+
+ * Documentation for the CLI converted to Asciidoc and moved to
+ http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
+ * Improved logging, and refactored a few methods to help with this.
+ * Dry-run output is now more like v2, with the index or snapshot in the log
+ line, along with the command. Several tests needed refactoring with this
+ change, along with a bit of documentation.
+
+**Bug fixes**
+
+ * Fix links to repository in setup.py. Reported in #318 (untergeek)
+ * No more ``--delay`` with optimized indices. Reported in #319 (untergeek)
+ * ``--request_timeout`` not working as expected. Reinstate the version 2
+ timeout override feature to prevent default timeouts for ``optimize`` and
+ ``snapshot`` operations. Reported in #320 (untergeek)
+ * Reduce index count to 200 for
+ test.integration.test_cli_commands.TestCLISnapshot.test_cli_snapshot_huge_list
+ in order to reduce or eliminate Jenkins CI test timeouts.
+ Reported in #324 (untergeek)
+ * ``--dry-run`` no longer calls ``show``, but will show output in the log, as
+ in v2. This was a recurring complaint. See #328 (untergeek)
+
+
+3.0.2 (23 Mar 2015)
+-------------------
+
+**Announcement**
+
+This is a bug fix release. #307 and #309 were big enough to warrant an
+expedited release.
+
+**Bug fixes**
+
+ * Purge unneeded constants, and clean up config options for snapshot. Reported in #303 (untergeek)
+ * Don't split large index list if performing snapshots. Reported in #307 (untergeek)
+ * Act correctly if a zero value for `--older-than` or `--newer-than` is provided. #309 (untergeek)
+
+3.0.1 (16 Mar 2015)
+-------------------
+
+**Announcement**
+
+The ``regex_iterate`` method was horribly named. It has been renamed to
+``apply_filter``. Methods have been added to allow API users to build a
+filtered list of indices similarly to how the CLI does. This was an oversight.
+Props to @SegFaultAX for pointing this out.
+
+**General**
+
+ * In conjunction with the rebrand to Elastic, URLs and documentation were updated.
+ * Renamed horribly named `regex_iterate` method to `apply_filter` #298 (untergeek)
+ * Added `build_filter` method to mimic CLI calls. #298 (untergeek)
+ * Added Examples page in the API documentation. #298 (untergeek)
+
+**Bug fixes**
+
+ * Refactored to show `--dry-run` info for `--disk-space` calls. Reported in
+ #290 (untergeek)
+ * Added list chunking so acting on huge lists of indices won't result in a URL
+ bigger than 4096 bytes (Elasticsearch's default limit.) Reported in
+ https://github.com/elastic/curator/issues/245#issuecomment-77916081
+ * Refactored `to_csv()` method to be simpler.
+ * Added and removed tests according to changes. Code coverage still at 99%
+
+3.0.0 (9 March 2015)
+--------------------
+
+**Release Notes**
+
+The full release of Curator 3.0 is out! Check out all of the changes here!
+
+*Note:* This release is _not_ reverse compatible with any previous version.
+
+Because 3.0 is a major point release, there have been some major changes to both
+the API as well as the CLI arguments and structure.
+
+Be sure to read the updated command-line specific docs in the
+[wiki](https://github.com/elasticsearch/curator/wiki) and change your
+command-line arguments accordingly.
+
... 991 lines suppressed ...
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/python-modules/packages/elasticsearch-curator.git
More information about the Python-modules-commits
mailing list