DuckCorp » DuckCorp Infrastructure

For wikitrans, here is the diff:

@@ -1025,7 +1025,7 @@
                 self.file = open(keywords[kw])
             elif kw == 'text':
                 if sys.version_info[0] > 2:
-                    self.text = keywords[kw].decode('utf-8').split("\n")
+                    self.text = keywords[kw].split("\n")
                 else:
                     self.text = keywords[kw].split("\n")
             elif kw == 'lang':

Without it we get this trace:

May 08 20:10:45 Jinta dicod[25119]: Error: Traceback (most recent call last):
May 08 20:10:45 Jinta dicod[25119]: Error:   File "/usr/share/dico/python/mediawiki.py", line 98, in define_word
May 08 20:10:45 Jinta dicod[25119]: Error:     wikiparser = TextWiktionaryMarkup (text=data)
May 08 20:10:45 Jinta dicod[25119]: Error:   File "/usr/lib/python3/dist-packages/wikitrans/wiki2text.py", line 278, in __init__
May 08 20:10:45 Jinta dicod[25119]: Error:     super(TextWikiMarkup, self).__init__(*args, **keywords)
May 08 20:10:45 Jinta dicod[25119]: Error:   File "/usr/lib/python3/dist-packages/wikitrans/wikimarkup.py", line 1028, in __init__
May 08 20:10:45 Jinta dicod[25119]: Error:     self.text = keywords[kw].decode('utf-8').split("\n")
May 08 20:10:45 Jinta dicod[25119]: Error: AttributeError: 'str' object has no attribute 'decode'

It seems the decoding already happens before.

Interestingly if I use the wikitrans program with a WP URL the original code works but the fix breaks with:

# wikitrans   https://en.wikipedia.org/wiki/Special:Export/Coin
Traceback (most recent call last):
  File "/usr/bin/wikitrans", line 186, in <module>
    main()
  File "/usr/bin/wikitrans", line 175, in main
    markup = handlers[options.otype][options.itype](**options.kwdict)
  File "/usr/lib/python3/dist-packages/wikitrans/wiki2html.py", line 241, in __init__
    super(HtmlWikiMarkup, self).__init__(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/wikitrans/wikimarkup.py", line 1029, in __init__
    self.text = keywords[kw].split("\n")
TypeError: a bytes-like object is required, not 'str'

I looked into the dicod mediawiki plugin and stumbled onto my own patch for Python 3 :-).

I made this patch witch works with the original wikitrans code:

--- /usr/share/dico/python/mediawiki.py.orig    2019-05-08 20:45:22.746677470 +0200
+++ /usr/share/dico/python/mediawiki.py    2019-05-08 20:45:11.702138585 +0200
@@ -93,8 +93,7 @@
             data = el[0].firstChild.data
             if dico.current_markup () != 'wiki':
                 data = self.__htmlentitydecode (data)
-                if sys.version_info[0] == 2:
-                    data = data.encode ('utf-8')
+                data = data.encode ('utf-8')
                 wikiparser = TextWiktionaryMarkup (text=data)
                 wikiparser.parse ()
                 data = str (wikiparser)

So I believe either wikitrans should be more flexible, or this patch should be applied in dicod.

As for ruby-mail, here is the diff:

@@ -183,10 +183,8 @@
         params = params.map { |i| i.split(/\s*\=\s*/) }
         params = params.map { |i| "#{i[0]}=#{dquote(i[1].to_s.gsub(/;$/,""))}" }.join('; ')
         "#{type}; #{params}" 
-      when val =~ /^\s*$/
-        'text/plain'
       else
-        ''
+        'text/plain; charset=us-ascii'
       end
     end

@@ -195,7 +193,7 @@
       when val =~ /^([\w\-]+)\/([\w\-]+);.+$/i
         "#{$1}/#{$2}" 
       else
-        'text/plain'
+        'text/plain; charset=us-ascii'
       end
     end
   end

The version in Buster has slightly changed but the logic is equally broken and the patch is still needed.

ruby-mail is blocked at 2.6.4 to avoid breaking Banya (which is probably broken anyway). I need to evaluate 2.7.1 if we decide to keep it.

The patch for redmine activities (thanks Pilou, see #655):

--- /dev/null   2015-12-14 19:43:03.976709288 +0100
+++ app/views/activities/index.api.rsb  2016-07-07 11:36:32.000000000 +0200
@@ -0,0 +1,13 @@
+api.array :activities do
+  @events.uniq.each do |event|
+    api.activity do
+      api.id          "#{event.class.name}/#{event.id}" 
+      api.title       event.event_title
+      api.type        event.event_type
+      api.project(:id => event.project.id, :name => event.project.name) unless event.project.nil?
+      api.description event.event_description
+      api.url         url_for(event.event_url)
+      api.created_on  event.event_datetime
+    end
+  end
+end
--- ./app/controllers/activities_controller.rb.orig     2016-10-10 09:48:13.000000000 +0200
+++ ./app/controllers/activities_controller.rb  2017-07-19 20:22:19.000000000 +0200
@@ -54,6 +54,7 @@
     end

     events = @activity.events(@date_from, @date_to)
+    @events = events # for API

     if events.empty? || stale?(:etag => [@activity.scope, @date_to, @date_from, @with_subprojects, @author, events.first, events.size, User.current, current_language])
       respond_to do |format|
@@ -61,6 +62,7 @@
           @events_by_day = events.group_by {|event| User.current.time_to_date(event.event_datetime)}
           render :layout => false if request.xhr?
         }
+        format.api
         format.atom {
           title = l(:label_activity)
           if @author

I found another patch repo_controller.patch but it's obsolete and the corresponding files were not found modified before switchign to Buster so this confirms it is no longer needed.

The wikitrans patch caused this error:

Apr 22 13:11:41 Jinta dicod[14411]: Info: Client info: "GNU Dico (Python Edition) 1.1" 
Apr 22 13:11:41 Jinta dicod[14411]: Error: Traceback (most recent call last):
Apr 22 13:11:41 Jinta dicod[14411]: Error:   File "/usr/share/dico/python/mediawiki.py", line 97, in define_word
Apr 22 13:11:41 Jinta dicod[14411]: Error:     wikiparser = TextWiktionaryMarkup (text=data)
Apr 22 13:11:41 Jinta dicod[14411]: Error:   File "/usr/lib/python3/dist-packages/wikitrans/wiki2text.py", line 278, in __init__
Apr 22 13:11:41 Jinta dicod[14411]: Error:     super(TextWikiMarkup, self).__init__(*args, **keywords)
Apr 22 13:11:41 Jinta dicod[14411]: Error:   File "/usr/lib/python3/dist-packages/wikitrans/wikimarkup.py", line 1028, in __init__
Apr 22 13:11:41 Jinta dicod[14411]: Error:     self.text = keywords[kw].decode('utf-8').split("\n")
Apr 22 13:11:41 Jinta dicod[14411]: Error: AttributeError: 'str' object has no attribute 'decode'
Apr 22 13:11:41 Jinta dicod[14375]: Debug: 14411 exited successfully

I unapplied it and it's works so far. I cannot even reproduce the problems that made me add it. Also from a Python 3 point of view I think this does not make sense to use decode().

I'll be monitoring the situation.

I updated the debsum-ignore in order to use regexes (ea6aa17724f60fddb2d3537b948a412f02bb1198).